Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.644160615s FAIL (2025-07-02 19:24:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.644188318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.601966983s FAIL (2025-07-02 19:23:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.60199613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.713316851s FAIL (2025-07-02 19:23:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.713349703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.642083914s FAIL (2025-07-02 19:22:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.642117106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.599948202s FAIL (2025-07-02 19:21:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.599976556s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.613491882s FAIL (2025-07-02 19:21:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.613530185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.630591664s FAIL (2025-07-02 19:20:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.630635828s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.919980702s FAIL (2025-07-02 19:19:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.920006731s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.668339714s FAIL (2025-07-02 19:19:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.668360163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.640498869s FAIL (2025-07-02 19:18:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.64052546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.665299808s FAIL (2025-07-02 19:18:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.665325418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.736016321s FAIL (2025-07-02 19:17:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.736046087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.719403916s FAIL (2025-07-02 19:16:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.719433603s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.729655212s FAIL (2025-07-02 19:16:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.729683086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.635540544s FAIL (2025-07-02 19:15:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.635572374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.708607232s FAIL (2025-07-02 19:14:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.708638943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.660911202s FAIL (2025-07-02 19:14:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.6609411s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.645777599s FAIL (2025-07-02 19:13:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.645802576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.661492662s FAIL (2025-07-02 19:13:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.661518061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.687301844s FAIL (2025-07-02 19:12:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.687333393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.686124028s FAIL (2025-07-02 19:12:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.686151051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.598534545s FAIL (2025-07-02 19:11:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.59856765s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.044059284s FAIL (2025-07-02 19:10:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 6.044100503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.723996622s FAIL (2025-07-02 19:09:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.724019394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.746275777s FAIL (2025-07-02 19:09:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.74630847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.658632512s FAIL (2025-07-02 19:08:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.658663692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.715726842s FAIL (2025-07-02 19:07:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.715752442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.712422332s FAIL (2025-07-02 19:06:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.71244808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.685787195s FAIL (2025-07-02 19:05:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.685821651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.68276831s FAIL (2025-07-02 19:04:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.682798467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.730678558s FAIL (2025-07-02 19:04:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.730704046s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.916594921s FAIL (2025-07-02 19:03:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.916622485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.909417768s FAIL (2025-07-02 19:02:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.909443477s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.631405419s FAIL (2025-07-02 19:01:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.63143263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.680014133s FAIL (2025-07-02 19:01:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.68004409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 22.664672525s FAIL (2025-07-02 19:00:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 22.664687564s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.242962789s FAIL (2025-07-02 19:00:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tItem_SegmentCommit Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments... Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil> Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments... Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment... Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment. Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece... Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments... Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil> Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments... Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 30.243037572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.652294122s FAIL (2025-07-02 17:38:55 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.652319581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.642279754s FAIL (2025-07-02 17:38:40 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.642311585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.640566919s FAIL (2025-07-02 17:38:18 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.640597307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.635660373s FAIL (2025-07-02 17:37:45 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.635705991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.71394807s FAIL (2025-07-02 17:37:25 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.713970534s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.671494585s FAIL (2025-07-02 17:36:43 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.671508911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.665072447s FAIL (2025-07-02 17:35:54 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.66509503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.644903259s FAIL (2025-07-02 17:35:07 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.644930473s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.471820401s FAIL (2025-07-02 17:34:42 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.471843997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.569740474s FAIL (2025-07-02 17:34:26 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.56978007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.676011294s FAIL (2025-07-02 17:34:00 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.67603967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.610575969s FAIL (2025-07-02 17:33:01 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.610601738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.636928353s FAIL (2025-07-02 17:32:14 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== l 2 16:53:34.097254581 (223h43m8.867895916s) [noise=true , address=77.90.31.13:4628 , piece_id=XMRK6IRQB7JXEMN7CXJOJHXUWZTOPOULFNYXOQIPLJHZD3NUBBVQ, seg_pos={0 0}, upload=178933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 2 16:53:34.097307081 (223h43m8.867948416s) [seg_pos={0 0}, upload=178933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:53:34.097386623 (223h43m8.868027959s) [upload=178933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:53:34.097392004 (223h43m8.868033329s) [upload=178933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:53:34.419688729 (223h43m9.190330064s) [upload=178933]: Done uploading segments... Jul 2 16:53:34.419709358 (223h43m9.190350703s) [upload=178933]: Upload finished. err: <nil> Jul 2 16:54:05.262461990 (223h43m40.033103325s) [upload=178943]: Starting upload Jul 2 16:54:05.262511685 (223h43m40.03315302s) [upload=178943]: Done starting upload Jul 2 16:54:05.262549186 (223h43m40.033190521s) [upload=178943]: Uploading segments... Jul 2 16:54:05.262676380 (223h43m40.033317715s) [upload=178943]: Got next segment. Inline: false Jul 2 16:54:05.262704774 (223h43m40.033346109s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 16:54:05.262708992 (223h43m40.033350327s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 16:54:05.512996024 (223h43m40.283637359s) [seg_pos={0 0}, upload=178943]: Begin upload segment... Jul 2 16:54:05.513035330 (223h43m40.283676665s) [seg_pos={0 0}, upload=178943]: Done begin upload segment. Jul 2 16:54:05.513051209 (223h43m40.283692544s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513061560 (223h43m40.283702895s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513394557 (223h43m40.284035892s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513703728 (223h43m40.284345053s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513773561 (223h43m40.284414896s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.513982151 (223h43m40.284623486s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514021205 (223h43m40.28466254s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514233051 (223h43m40.284874386s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514236677 (223h43m40.284878002s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514851123 (223h43m40.285492458s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.514883755 (223h43m40.28552509s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515120498 (223h43m40.285761823s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.515370256 (223h43m40.286011591s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516025430 (223h43m40.286666765s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.516534103 (223h43m40.287175438s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:05.517178174 (223h43m40.28781951s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Uploading piece... Jul 2 16:54:07.112928981 (223h43m41.883570316s) [upload=178943]: Next returned nil segment Jul 2 16:54:07.112946585 (223h43m41.88358792s) [upload=178943]: Waiting for error group managing segments... Jul 2 16:54:07.457575995 (223h43m42.22821733s) [noise=true , address=104.36.82.13:4186 , piece_id=RDUZYI4ZIMHJOVCPITSYXP3CSWDAHJOIEXO3MGCGERCVASMUJSFA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.458802101 (223h43m42.229443436s) [noise=true , address=104.36.82.10:4079 , piece_id=JJKSDFXZOD2ENTWHHP6SUZEAAZGNNN4IM666NJ4WCV2CM227H72A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.466429934 (223h43m42.237071269s) [noise=true , address=104.36.82.11:4115 , piece_id=TPUNGN5NPZJQJBT4UZLIZSUT5HOHPGJIIFFWZJXJS5EVIQ23NKHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.474520202 (223h43m42.245161527s) [noise=true , address=104.36.82.9:4006 , piece_id=2UAR4BOSXE6CGDL4ABPEMO7MPWD2OWYAOMRJ37UNZJAF6SKZZ6BA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.488959054 (223h43m42.259600389s) [noise=true , address=104.36.82.14:4222 , piece_id=7TZ6E5X756SYA2HBJS7YKHHEN2XPZOQBLLOZFCKILN3EPROGNGOA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.491690220 (223h43m42.262331555s) [noise=true , address=104.36.82.8:4972 , piece_id=DFR3IP4T4MDTVGF5V6P6VE5SY2FG5XOSSNNH3RIUJ6AXFPQSEZQA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.503267458 (223h43m42.273908793s) [noise=true , address=104.36.82.7:4912 , piece_id=BZREKTJMJB7HSVRVYZAMZFQKXSK6MWDPYMJAKWOBTVM5HO5NLPRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.514613936 (223h43m42.285255271s) [noise=true , address=104.36.82.15:4259 , piece_id=RYBTGQXRTINIJQCGSZMMIEXFS6Q2I5A7IXZUOREK3TOY2T3BEPRQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.531889065 (223h43m42.3025304s) [noise=true , address=104.36.82.7:4911 , piece_id=SUQQYFCXDQTI7HCZDHR53SFRNAAE3PZYINZLOT6NYNWAHXGHTJNQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.582720442 (223h43m42.353361767s) [noise=true , address=104.36.82.4:4801 , piece_id=NWH42PADWTEBGLML25LBC7N4LXJWL4F2G3EISFJFFWNB2GCN7NHA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.584813677 (223h43m42.355455012s) [noise=true , address=45.91.116.6:4973 , piece_id=LZ527XSEHHZSQGIZEFTM7W7UUDRU7DY366V22DSIF3O7SVZETBJA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.607187571 (223h43m42.377828896s) [noise=true , address=77.90.31.19:4842 , piece_id=4AYWI756KXIJGDT57RIY7HXL6DLNOJKHDJPXZVEGEBCEQCWYBIHQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608616895 (223h43m42.37925824s) [noise=true , address=77.90.31.16:4734 , piece_id=6BN6SSHMCDDPZLVVJ6CAWQV5FFAHNFIQAULDMMYI63JLDO5HVODQ, seg_pos={0 0}, upload=178943]: Done uploading piece. err: <nil> Jul 2 16:54:07.608631964 (223h43m42.379273299s) [seg_pos={0 0}, upload=178943]: Segment reached optimal threshold of 13 pieces. Jul 2 16:54:07.608785709 (223h43m42.379427034s) [noise=true , address=45.91.116.5:4297 , piece_id=CQHM6T74GPL6JIEZ5YYZLBVESYT3O2A76CZJCKV3NGRSOR5BMV4Q, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608881211 (223h43m42.379522546s) [noise=true , address=45.91.116.12:4226 , piece_id=7XR3OU6KKUINFBTHVLYLHTYXKS5TL3MQ4G46GN7UJMKGA52K6JRA, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.608968247 (223h43m42.379609572s) [noise=true , address=45.91.116.8:4045 , piece_id=6GK64WE6N372L2GR34EJIUC7OQUXWAIBNLS5MU3B2M2EPV75Q65A, seg_pos={0 0}, upload=178943]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Jul 2 16:54:07.609004217 (223h43m42.379645552s) [seg_pos={0 0}, upload=178943]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 2 16:54:07.609097285 (223h43m42.37973862s) [upload=178943]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 16:54:07.609104839 (223h43m42.379746174s) [upload=178943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 16:54:07.995690087 (223h43m42.766331422s) [upload=178943]: Done uploading segments... Jul 2 16:54:07.995712629 (223h43m42.766353974s) [upload=178943]: Upload finished. err: <nil> Duration: 5.636954072s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)