Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...........X.X..X..X........X..X.....X.X.X.X..X.X..X.....XX.] duration: 6.155111072s trace: https://jaeger.dev.storj.io/trace/4227f12f1c6df6e7 FAIL (2025-12-29 06:35:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 90]: Starting upload Dec 29 06:34:32.524251360 (567h5m32.395334395s) [upload=497190]: Done starting upload Dec 29 06:34:32.524308078 (567h5m32.395391114s) [upload=497190]: Uploading segments... Dec 29 06:34:32.524442545 (567h5m32.39552558s) [upload=497190]: Got next segment. Inline: false Dec 29 06:34:32.524473784 (567h5m32.395556819s) [upload=497190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:34:32.524477962 (567h5m32.395560997s) [upload=497190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:34:32.775681697 (567h5m32.646764732s) [seg_pos={0 0}, upload=497190]: Begin upload segment... Dec 29 06:34:32.775729198 (567h5m32.646812243s) [seg_pos={0 0}, upload=497190]: Done begin upload segment. Dec 29 06:34:32.775759054 (567h5m32.646842099s) [noise=true , address=145.79.227.50:4373 , piece_id=HNL3HTYTWTRKT4FJPRUCUBYUZLR64D4PO5ZGTMXKCNRAMTXAZQWA, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.776881803 (567h5m32.647964838s) [noise=true , address=45.91.116.4:4939 , piece_id=2UQI377HWHTUBUMGKRNOBAVCHCM5FOCFU5FHVZAPLTTRLO2H3QNA, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.777311180 (567h5m32.648394215s) [noise=true , address=45.91.116.6:4975 , piece_id=L6SDCU7Y4OTRSRWA646PAB2EUAWGA3ZA75UVY7GBC5AQL54YPEEQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.777594449 (567h5m32.648677484s) [noise=true , address=45.91.116.11:4117 , piece_id=AKGXLGV4FAJQTQPSYXXQNH6BJTHVEXWUSPK7TYDBK5DFV6W2IUMQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.777896274 (567h5m32.648979319s) [noise=true , address=45.91.116.7:4009 , piece_id=CXWVAURHLEBH4RB6C524JSEMBD5LGLGK3IRRU3SPRRIIPWD6IPHQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.778221153 (567h5m32.649304188s) [noise=true , address=45.91.116.14:4192 , piece_id=CGTMNSQPOBUIDTPHVROSSCU7SDPQTHNVASILYV7MNBFRGDZJ2QFQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.778637776 (567h5m32.649720821s) [noise=true , address=145.79.227.14:4837 , piece_id=FUOSCXXRJVLYDIPVSERY4DAHLOQKKTYQTEJ7JEA7GJ32I72HATLQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.779052656 (567h5m32.650135701s) [noise=true , address=145.79.227.46:4602 , piece_id=7AN7P5LGKM4KFX7ZWMXXAMLY43WYT5BBQZO33GF4TTNAUDI6W43Q, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.779372685 (567h5m32.65045573s) [noise=true , address=145.79.227.19:4078 , piece_id=AZONXAFTAAUGVEHNYVHPDQCNAXYQ2XIN4ZRNYADRCIZOK36T4OXQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.779737541 (567h5m32.650820576s) [noise=true , address=145.79.227.48:4302 , piece_id=ZO2JK2VCQVHBB4FLIEZJBYHDOXJR3WPNZF2NSBEA22O3FCTXYDOQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.780870057 (567h5m32.651953092s) [noise=true , address=145.79.227.18:4007 , piece_id=OMBIZLWEIOTGQTQ7ISPMTTBEWGUP7N5KZMPN42PZ34CD6PIJLHXA, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.781962487 (567h5m32.653045522s) [noise=true , address=45.91.116.12:4225 , piece_id=E3IQMWKRQY2HJI3BPHBNN3QFIYST7JTAAZ3NJBSKHOOQIGXYKIDQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.783104932 (567h5m32.654187977s) [noise=true , address=77.90.31.10:4518 , piece_id=C4AKCR5KFHFMP3MMSI6ECKNBKN2K2LQDTPYBC7LOBU6WORVIEWDQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.784227320 (567h5m32.655310365s) [noise=true , address=145.79.227.39:4014 , piece_id=IFLHBVV5ZN7MBNESXLYSUKVGKHWG5YJD43NTJV7NWLBKZJNAETJA, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.785361279 (567h5m32.656444324s) [noise=true , address=45.91.116.10:4081 , piece_id=JLLW7TQJB2RZNFTUQ24Z3DFYLNPHXTYNOYC4PMHWO5PBKY4SGEIA, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.786740405 (567h5m32.65782344s) [noise=true , address=145.79.227.15:4875 , piece_id=POYDXEQDNG527JPMHCMAFRYM7SZ3ISIFPKS2HEUQPDIBSF5G7O7Q, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:34.475431594 (567h5m34.346514629s) [upload=497190]: Next returned nil segment Dec 29 06:34:34.475449679 (567h5m34.346532714s) [upload=497190]: Waiting for error group managing segments... Dec 29 06:34:34.760279684 (567h5m34.631362719s) [noise=true , address=145.79.227.46:4602 , piece_id=7AN7P5LGKM4KFX7ZWMXXAMLY43WYT5BBQZO33GF4TTNAUDI6W43Q, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.764142299 (567h5m34.635225334s) [noise=true , address=145.79.227.39:4014 , piece_id=IFLHBVV5ZN7MBNESXLYSUKVGKHWG5YJD43NTJV7NWLBKZJNAETJA, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.765221234 (567h5m34.636304279s) [noise=true , address=145.79.227.19:4078 , piece_id=AZONXAFTAAUGVEHNYVHPDQCNAXYQ2XIN4ZRNYADRCIZOK36T4OXQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.777367142 (567h5m34.648450177s) [noise=true , address=145.79.227.18:4007 , piece_id=OMBIZLWEIOTGQTQ7ISPMTTBEWGUP7N5KZMPN42PZ34CD6PIJLHXA, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.793754145 (567h5m34.66483718s) [noise=true , address=145.79.227.15:4875 , piece_id=POYDXEQDNG527JPMHCMAFRYM7SZ3ISIFPKS2HEUQPDIBSF5G7O7Q, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.794152153 (567h5m34.665235188s) [noise=true , address=145.79.227.48:4302 , piece_id=ZO2JK2VCQVHBB4FLIEZJBYHDOXJR3WPNZF2NSBEA22O3FCTXYDOQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.795769122 (567h5m34.666852167s) [noise=true , address=145.79.227.50:4373 , piece_id=HNL3HTYTWTRKT4FJPRUCUBYUZLR64D4PO5ZGTMXKCNRAMTXAZQWA, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.993123118 (567h5m34.864206164s) [noise=true , address=45.91.116.4:4939 , piece_id=2UQI377HWHTUBUMGKRNOBAVCHCM5FOCFU5FHVZAPLTTRLO2H3QNA, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.032563819 (567h5m34.903646864s) [noise=true , address=45.91.116.6:4975 , piece_id=L6SDCU7Y4OTRSRWA646PAB2EUAWGA3ZA75UVY7GBC5AQL54YPEEQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.066847582 (567h5m34.937930617s) [noise=true , address=45.91.116.10:4081 , piece_id=JLLW7TQJB2RZNFTUQ24Z3DFYLNPHXTYNOYC4PMHWO5PBKY4SGEIA, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.150526631 (567h5m35.021609676s) [noise=true , address=45.91.116.12:4225 , piece_id=E3IQMWKRQY2HJI3BPHBNN3QFIYST7JTAAZ3NJBSKHOOQIGXYKIDQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.181132701 (567h5m35.052215736s) [noise=true , address=45.91.116.7:4009 , piece_id=CXWVAURHLEBH4RB6C524JSEMBD5LGLGK3IRRU3SPRRIIPWD6IPHQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.273236191 (567h5m35.144319236s) [noise=true , address=77.90.31.10:4518 , piece_id=C4AKCR5KFHFMP3MMSI6ECKNBKN2K2LQDTPYBC7LOBU6WORVIEWDQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.273256070 (567h5m35.144339105s) [seg_pos={0 0}, upload=497190]: Segment reached optimal threshold of 13 pieces. Dec 29 06:34:35.273502970 (567h5m35.144586005s) [noise=true , address=45.91.116.11:4117 , piece_id=AKGXLGV4FAJQTQPSYXXQNH6BJTHVEXWUSPK7TYDBK5DFV6W2IUMQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:50580->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:50580->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50580->45.91.116.11:4117: use of closed network connection Dec 29 06:34:35.273509672 (567h5m35.144592707s) [noise=true , address=45.91.116.11:4117 , piece_id=AKGXLGV4FAJQTQPSYXXQNH6BJTHVEXWUSPK7TYDBK5DFV6W2IUMQ, seg_pos={0 0}, upload=497190]: Long tail cancellation due to optimal threshold Dec 29 06:34:35.273789575 (567h5m35.14487261s) [noise=true , address=145.79.227.14:4837 , piece_id=FUOSCXXRJVLYDIPVSERY4DAHLOQKKTYQTEJ7JEA7GJ32I72HATLQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 29 06:34:35.273833659 (567h5m35.144916694s) [noise=true , address=145.79.227.14:4837 , piece_id=FUOSCXXRJVLYDIPVSERY4DAHLOQKKTYQTEJ7JEA7GJ32I72HATLQ, seg_pos={0 0}, upload=497190]: Long tail cancellation due to optimal threshold Dec 29 06:34:35.274074798 (567h5m35.145157833s) [noise=true , address=45.91.116.14:4192 , piece_id=CGTMNSQPOBUIDTPHVROSSCU7SDPQTHNVASILYV7MNBFRGDZJ2QFQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:36730->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 06:34:35.274082232 (567h5m35.145165267s) [noise=true , address=45.91.116.14:4192 , piece_id=CGTMNSQPOBUIDTPHVROSSCU7SDPQTHNVASILYV7MNBFRGDZJ2QFQ, seg_pos={0 0}, upload=497190]: Long tail cancellation due to optimal threshold Dec 29 06:34:35.274098814 (567h5m35.145181859s) [seg_pos={0 0}, upload=497190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:34:35.274168807 (567h5m35.145251842s) [upload=497190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:34:35.274173506 (567h5m35.145256541s) [upload=497190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:34:35.573661823 (567h5m35.444744858s) [upload=497190]: Done uploading segments... Dec 29 06:34:35.573716819 (567h5m35.444799854s) [upload=497190]: Upload finished. err: <nil> Duration: 6.155137542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.....................X....................................] duration: 21.973197125s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:35:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 21.97322033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....XX......................X..XX................X..........] duration: 4.001817239s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:35:11 UTC) orbiter warning: context deadline exceeded Duration: 4.001844111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X...........X.X..X..X........X..X.....X.X.X.X..X.X..X.....XX] duration: 30.004794738s trace: https://jaeger.dev.storj.io/trace/4227f12f1c6df6e7 FAIL (2025-12-29 06:35:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 90]: Starting upload Dec 29 06:34:32.524251360 (567h5m32.395334395s) [upload=497190]: Done starting upload Dec 29 06:34:32.524308078 (567h5m32.395391114s) [upload=497190]: Uploading segments... Dec 29 06:34:32.524442545 (567h5m32.39552558s) [upload=497190]: Got next segment. Inline: false Dec 29 06:34:32.524473784 (567h5m32.395556819s) [upload=497190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:34:32.524477962 (567h5m32.395560997s) [upload=497190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:34:32.775681697 (567h5m32.646764732s) [seg_pos={0 0}, upload=497190]: Begin upload segment... Dec 29 06:34:32.775729198 (567h5m32.646812243s) [seg_pos={0 0}, upload=497190]: Done begin upload segment. Dec 29 06:34:32.775759054 (567h5m32.646842099s) [noise=true , address=145.79.227.50:4373 , piece_id=HNL3HTYTWTRKT4FJPRUCUBYUZLR64D4PO5ZGTMXKCNRAMTXAZQWA, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.776881803 (567h5m32.647964838s) [noise=true , address=45.91.116.4:4939 , piece_id=2UQI377HWHTUBUMGKRNOBAVCHCM5FOCFU5FHVZAPLTTRLO2H3QNA, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.777311180 (567h5m32.648394215s) [noise=true , address=45.91.116.6:4975 , piece_id=L6SDCU7Y4OTRSRWA646PAB2EUAWGA3ZA75UVY7GBC5AQL54YPEEQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.777594449 (567h5m32.648677484s) [noise=true , address=45.91.116.11:4117 , piece_id=AKGXLGV4FAJQTQPSYXXQNH6BJTHVEXWUSPK7TYDBK5DFV6W2IUMQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.777896274 (567h5m32.648979319s) [noise=true , address=45.91.116.7:4009 , piece_id=CXWVAURHLEBH4RB6C524JSEMBD5LGLGK3IRRU3SPRRIIPWD6IPHQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.778221153 (567h5m32.649304188s) [noise=true , address=45.91.116.14:4192 , piece_id=CGTMNSQPOBUIDTPHVROSSCU7SDPQTHNVASILYV7MNBFRGDZJ2QFQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.778637776 (567h5m32.649720821s) [noise=true , address=145.79.227.14:4837 , piece_id=FUOSCXXRJVLYDIPVSERY4DAHLOQKKTYQTEJ7JEA7GJ32I72HATLQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.779052656 (567h5m32.650135701s) [noise=true , address=145.79.227.46:4602 , piece_id=7AN7P5LGKM4KFX7ZWMXXAMLY43WYT5BBQZO33GF4TTNAUDI6W43Q, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.779372685 (567h5m32.65045573s) [noise=true , address=145.79.227.19:4078 , piece_id=AZONXAFTAAUGVEHNYVHPDQCNAXYQ2XIN4ZRNYADRCIZOK36T4OXQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.779737541 (567h5m32.650820576s) [noise=true , address=145.79.227.48:4302 , piece_id=ZO2JK2VCQVHBB4FLIEZJBYHDOXJR3WPNZF2NSBEA22O3FCTXYDOQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.780870057 (567h5m32.651953092s) [noise=true , address=145.79.227.18:4007 , piece_id=OMBIZLWEIOTGQTQ7ISPMTTBEWGUP7N5KZMPN42PZ34CD6PIJLHXA, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.781962487 (567h5m32.653045522s) [noise=true , address=45.91.116.12:4225 , piece_id=E3IQMWKRQY2HJI3BPHBNN3QFIYST7JTAAZ3NJBSKHOOQIGXYKIDQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.783104932 (567h5m32.654187977s) [noise=true , address=77.90.31.10:4518 , piece_id=C4AKCR5KFHFMP3MMSI6ECKNBKN2K2LQDTPYBC7LOBU6WORVIEWDQ, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.784227320 (567h5m32.655310365s) [noise=true , address=145.79.227.39:4014 , piece_id=IFLHBVV5ZN7MBNESXLYSUKVGKHWG5YJD43NTJV7NWLBKZJNAETJA, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.785361279 (567h5m32.656444324s) [noise=true , address=45.91.116.10:4081 , piece_id=JLLW7TQJB2RZNFTUQ24Z3DFYLNPHXTYNOYC4PMHWO5PBKY4SGEIA, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:32.786740405 (567h5m32.65782344s) [noise=true , address=145.79.227.15:4875 , piece_id=POYDXEQDNG527JPMHCMAFRYM7SZ3ISIFPKS2HEUQPDIBSF5G7O7Q, seg_pos={0 0}, upload=497190]: Uploading piece... Dec 29 06:34:34.475431594 (567h5m34.346514629s) [upload=497190]: Next returned nil segment Dec 29 06:34:34.475449679 (567h5m34.346532714s) [upload=497190]: Waiting for error group managing segments... Dec 29 06:34:34.760279684 (567h5m34.631362719s) [noise=true , address=145.79.227.46:4602 , piece_id=7AN7P5LGKM4KFX7ZWMXXAMLY43WYT5BBQZO33GF4TTNAUDI6W43Q, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.764142299 (567h5m34.635225334s) [noise=true , address=145.79.227.39:4014 , piece_id=IFLHBVV5ZN7MBNESXLYSUKVGKHWG5YJD43NTJV7NWLBKZJNAETJA, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.765221234 (567h5m34.636304279s) [noise=true , address=145.79.227.19:4078 , piece_id=AZONXAFTAAUGVEHNYVHPDQCNAXYQ2XIN4ZRNYADRCIZOK36T4OXQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.777367142 (567h5m34.648450177s) [noise=true , address=145.79.227.18:4007 , piece_id=OMBIZLWEIOTGQTQ7ISPMTTBEWGUP7N5KZMPN42PZ34CD6PIJLHXA, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.793754145 (567h5m34.66483718s) [noise=true , address=145.79.227.15:4875 , piece_id=POYDXEQDNG527JPMHCMAFRYM7SZ3ISIFPKS2HEUQPDIBSF5G7O7Q, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.794152153 (567h5m34.665235188s) [noise=true , address=145.79.227.48:4302 , piece_id=ZO2JK2VCQVHBB4FLIEZJBYHDOXJR3WPNZF2NSBEA22O3FCTXYDOQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.795769122 (567h5m34.666852167s) [noise=true , address=145.79.227.50:4373 , piece_id=HNL3HTYTWTRKT4FJPRUCUBYUZLR64D4PO5ZGTMXKCNRAMTXAZQWA, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:34.993123118 (567h5m34.864206164s) [noise=true , address=45.91.116.4:4939 , piece_id=2UQI377HWHTUBUMGKRNOBAVCHCM5FOCFU5FHVZAPLTTRLO2H3QNA, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.032563819 (567h5m34.903646864s) [noise=true , address=45.91.116.6:4975 , piece_id=L6SDCU7Y4OTRSRWA646PAB2EUAWGA3ZA75UVY7GBC5AQL54YPEEQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.066847582 (567h5m34.937930617s) [noise=true , address=45.91.116.10:4081 , piece_id=JLLW7TQJB2RZNFTUQ24Z3DFYLNPHXTYNOYC4PMHWO5PBKY4SGEIA, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.150526631 (567h5m35.021609676s) [noise=true , address=45.91.116.12:4225 , piece_id=E3IQMWKRQY2HJI3BPHBNN3QFIYST7JTAAZ3NJBSKHOOQIGXYKIDQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.181132701 (567h5m35.052215736s) [noise=true , address=45.91.116.7:4009 , piece_id=CXWVAURHLEBH4RB6C524JSEMBD5LGLGK3IRRU3SPRRIIPWD6IPHQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.273236191 (567h5m35.144319236s) [noise=true , address=77.90.31.10:4518 , piece_id=C4AKCR5KFHFMP3MMSI6ECKNBKN2K2LQDTPYBC7LOBU6WORVIEWDQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: <nil> Dec 29 06:34:35.273256070 (567h5m35.144339105s) [seg_pos={0 0}, upload=497190]: Segment reached optimal threshold of 13 pieces. Dec 29 06:34:35.273502970 (567h5m35.144586005s) [noise=true , address=45.91.116.11:4117 , piece_id=AKGXLGV4FAJQTQPSYXXQNH6BJTHVEXWUSPK7TYDBK5DFV6W2IUMQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:50580->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:50580->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50580->45.91.116.11:4117: use of closed network connection Dec 29 06:34:35.273509672 (567h5m35.144592707s) [noise=true , address=45.91.116.11:4117 , piece_id=AKGXLGV4FAJQTQPSYXXQNH6BJTHVEXWUSPK7TYDBK5DFV6W2IUMQ, seg_pos={0 0}, upload=497190]: Long tail cancellation due to optimal threshold Dec 29 06:34:35.273789575 (567h5m35.14487261s) [noise=true , address=145.79.227.14:4837 , piece_id=FUOSCXXRJVLYDIPVSERY4DAHLOQKKTYQTEJ7JEA7GJ32I72HATLQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 29 06:34:35.273833659 (567h5m35.144916694s) [noise=true , address=145.79.227.14:4837 , piece_id=FUOSCXXRJVLYDIPVSERY4DAHLOQKKTYQTEJ7JEA7GJ32I72HATLQ, seg_pos={0 0}, upload=497190]: Long tail cancellation due to optimal threshold Dec 29 06:34:35.274074798 (567h5m35.145157833s) [noise=true , address=45.91.116.14:4192 , piece_id=CGTMNSQPOBUIDTPHVROSSCU7SDPQTHNVASILYV7MNBFRGDZJ2QFQ, seg_pos={0 0}, upload=497190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:36730->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 06:34:35.274082232 (567h5m35.145165267s) [noise=true , address=45.91.116.14:4192 , piece_id=CGTMNSQPOBUIDTPHVROSSCU7SDPQTHNVASILYV7MNBFRGDZJ2QFQ, seg_pos={0 0}, upload=497190]: Long tail cancellation due to optimal threshold Dec 29 06:34:35.274098814 (567h5m35.145181859s) [seg_pos={0 0}, upload=497190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:34:35.274168807 (567h5m35.145251842s) [upload=497190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:34:35.274173506 (567h5m35.145256541s) [upload=497190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:34:35.573661823 (567h5m35.444744858s) [upload=497190]: Done uploading segments... Dec 29 06:34:35.573716819 (567h5m35.444799854s) [upload=497190]: Upload finished. err: <nil> Duration: 30.004820927s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....XX......................X..XX................X.........] duration: 3.458034565s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:34:32 UTC) orbiter warning: context deadline exceeded Duration: 3.458056246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X.....................X...................................] duration: 15.384117652s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:34:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 15.384150576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X...........X.X..X..X........X..X.....X.X.X.X..X.X..X.....X] duration: 30.004035906s trace: https://jaeger.dev.storj.io/trace/2c288e83b7b5552 FAIL (2025-12-29 06:33:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: MUTYHITJJ5MF7DQQ7OYWZWNZTM6G6LMHUUEYYYSHYSVTCBW6LR7Q) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SJQOJLSGWVRGGY5IGGRNF33UPWTZXZAUQGNZ2LGU3MYA3PVIOIKA) context deadline exceeded ======LOGS====== m29.959052833s) [upload=497173]: Uploading segments... Dec 29 06:33:30.088140202 (567h4m29.959223247s) [upload=497173]: Got next segment. Inline: false Dec 29 06:33:30.088181190 (567h4m29.959264225s) [upload=497173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:33:30.088234111 (567h4m29.959317156s) [upload=497173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:33:30.330408033 (567h4m30.201491068s) [seg_pos={0 0}, upload=497173]: Begin upload segment... Dec 29 06:33:30.330472505 (567h4m30.20155556s) [seg_pos={0 0}, upload=497173]: Done begin upload segment. Dec 29 06:33:30.330550574 (567h4m30.201633609s) [noise=true , address=145.79.227.47:4266 , piece_id=OOMOHGYW6MWEFSTBKDQ4B555VQLEDQIQFPDENXPI6SMCYHGRJAVA, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.330566113 (567h4m30.201649158s) [noise=true , address=145.79.227.38:4978 , piece_id=GOCE4TMXSJVTGJHDUJOLRCN3XN5K5RZZAA573APZD4A7UCBPRSIA, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.331216181 (567h4m30.202299216s) [noise=true , address=77.90.31.13:4626 , piece_id=MUTYHITJJ5MF7DQQ7OYWZWNZTM6G6LMHUUEYYYSHYSVTCBW6LR7Q, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.331361217 (567h4m30.202444252s) [noise=true , address=145.79.227.4:4185 , piece_id=KOSUY4KTUZNOWTGL357LL26GRHGLMNBDIADXNQQTJ7ZVZXCX6JJA, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.331930681 (567h4m30.203013726s) [noise=true , address=45.91.116.12:4226 , piece_id=SJQOJLSGWVRGGY5IGGRNF33UPWTZXZAUQGNZ2LGU3MYA3PVIOIKA, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.332192079 (567h4m30.203275125s) [noise=true , address=145.79.227.16:4911 , piece_id=DE2TGZA4H6BOCHNHORLVAZMHJWTTD4UNIO2BNXKWM45K7VXLRYQQ, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.332776092 (567h4m30.203859127s) [noise=true , address=45.91.116.11:4120 , piece_id=YJJPC264P2Z2K7OOKVILGFRFYADYCMATJ44JJNH6XOW52T64X7FQ, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.332903434 (567h4m30.203986479s) [noise=true , address=145.79.227.5:4223 , piece_id=5W6MFOM5OUVINXMSH3O6L65F4EP6RTSY66YZRFIQSGSQ6PQTPQXA, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.333600681 (567h4m30.204683716s) [noise=true , address=145.79.227.13:4803 , piece_id=DMW4OFG4EXLCZFFEH72GAT2E223NS52LYGSDRJHC6DXLNX4ZGWHA, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.333699359 (567h4m30.204782394s) [noise=true , address=145.79.227.2:4115 , piece_id=HPC3MSUIPCAJLVOKDPT6HNMXWRZI5P52OWIQKRZIB3J6XWR3VHIQ, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.334317276 (567h4m30.205400321s) [noise=true , address=45.91.116.10:4081 , piece_id=J4LKZPB6G5UCEAKZCFDVUAWS72EOLOTP77PNJZWQANN4KWF4HL4Q, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.334413900 (567h4m30.205496935s) [noise=true , address=145.79.227.20:4437 , piece_id=QDKYEN4QB2XTUP7EERBZI3LRAVFOPAVSH2TUFQPWWSBQ3GMYOFLQ, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.334996339 (567h4m30.206079374s) [noise=true , address=145.79.227.41:4086 , piece_id=EHQS2QJXMLDZQO4ANQLEOUBV2YRAW7R2E7ZQ7LQA46FVTALEL4LA, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.335052747 (567h4m30.206135782s) [noise=true , address=45.91.116.14:4189 , piece_id=35H3NRSKDADYR6OCD3JEANYW5KAP3YRPM2T4UX3BHCMHATVW4Q5Q, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.335676064 (567h4m30.206759099s) [noise=true , address=45.91.116.13:4154 , piece_id=JWOMKEEEAPZPZHPUIBSNPJZ3DK5LBDWOMMYXLZPUYOD7SZUPHMQQ, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:30.335804408 (567h4m30.206887443s) [noise=true , address=45.91.116.4:4937 , piece_id=D3TLTZCVJ7JFMSDO2UMCOBSFDS4NQP7KEBCA5YVXIXYF3VWKAUTQ, seg_pos={0 0}, upload=497173]: Uploading piece... Dec 29 06:33:32.029416328 (567h4m31.900499373s) [upload=497173]: Next returned nil segment Dec 29 06:33:32.029438480 (567h4m31.900521525s) [upload=497173]: Waiting for error group managing segments... Dec 29 06:33:32.321349198 (567h4m32.192432233s) [noise=true , address=145.79.227.4:4185 , piece_id=KOSUY4KTUZNOWTGL357LL26GRHGLMNBDIADXNQQTJ7ZVZXCX6JJA, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.324884359 (567h4m32.195967394s) [noise=true , address=145.79.227.41:4086 , piece_id=EHQS2QJXMLDZQO4ANQLEOUBV2YRAW7R2E7ZQ7LQA46FVTALEL4LA, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.325217153 (567h4m32.196300198s) [noise=true , address=145.79.227.16:4911 , piece_id=DE2TGZA4H6BOCHNHORLVAZMHJWTTD4UNIO2BNXKWM45K7VXLRYQQ, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.336310748 (567h4m32.207393793s) [noise=true , address=145.79.227.13:4803 , piece_id=DMW4OFG4EXLCZFFEH72GAT2E223NS52LYGSDRJHC6DXLNX4ZGWHA, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.337389231 (567h4m32.208472266s) [noise=true , address=145.79.227.20:4437 , piece_id=QDKYEN4QB2XTUP7EERBZI3LRAVFOPAVSH2TUFQPWWSBQ3GMYOFLQ, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.338197500 (567h4m32.209280535s) [noise=true , address=145.79.227.2:4115 , piece_id=HPC3MSUIPCAJLVOKDPT6HNMXWRZI5P52OWIQKRZIB3J6XWR3VHIQ, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.339143411 (567h4m32.210226446s) [noise=true , address=145.79.227.38:4978 , piece_id=GOCE4TMXSJVTGJHDUJOLRCN3XN5K5RZZAA573APZD4A7UCBPRSIA, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.343119612 (567h4m32.214202647s) [noise=true , address=145.79.227.5:4223 , piece_id=5W6MFOM5OUVINXMSH3O6L65F4EP6RTSY66YZRFIQSGSQ6PQTPQXA, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.351485753 (567h4m32.222568799s) [noise=true , address=145.79.227.47:4266 , piece_id=OOMOHGYW6MWEFSTBKDQ4B555VQLEDQIQFPDENXPI6SMCYHGRJAVA, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.501615957 (567h4m32.372699022s) [noise=true , address=77.90.31.13:4626 , piece_id=MUTYHITJJ5MF7DQQ7OYWZWNZTM6G6LMHUUEYYYSHYSVTCBW6LR7Q, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.577694854 (567h4m32.448777899s) [noise=true , address=45.91.116.14:4189 , piece_id=35H3NRSKDADYR6OCD3JEANYW5KAP3YRPM2T4UX3BHCMHATVW4Q5Q, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.632457512 (567h4m32.503540547s) [noise=true , address=45.91.116.12:4226 , piece_id=SJQOJLSGWVRGGY5IGGRNF33UPWTZXZAUQGNZ2LGU3MYA3PVIOIKA, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.912577058 (567h4m32.783660093s) [noise=true , address=45.91.116.10:4081 , piece_id=J4LKZPB6G5UCEAKZCFDVUAWS72EOLOTP77PNJZWQANN4KWF4HL4Q, seg_pos={0 0}, upload=497173]: Done uploading piece. err: <nil> Dec 29 06:33:32.912597106 (567h4m32.783680141s) [seg_pos={0 0}, upload=497173]: Segment reached optimal threshold of 13 pieces. Dec 29 06:33:32.912800002 (567h4m32.783883047s) [noise=true , address=45.91.116.13:4154 , piece_id=JWOMKEEEAPZPZHPUIBSNPJZ3DK5LBDWOMMYXLZPUYOD7SZUPHMQQ, seg_pos={0 0}, upload=497173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 06:33:32.912805914 (567h4m32.783888959s) [noise=true , address=45.91.116.13:4154 , piece_id=JWOMKEEEAPZPZHPUIBSNPJZ3DK5LBDWOMMYXLZPUYOD7SZUPHMQQ, seg_pos={0 0}, upload=497173]: Long tail cancellation due to optimal threshold Dec 29 06:33:32.912813749 (567h4m32.783896784s) [noise=true , address=45.91.116.4:4937 , piece_id=D3TLTZCVJ7JFMSDO2UMCOBSFDS4NQP7KEBCA5YVXIXYF3VWKAUTQ, seg_pos={0 0}, upload=497173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:32976->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:32976->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32976->45.91.116.4:4937: use of closed network connection Dec 29 06:33:32.912826573 (567h4m32.783909648s) [noise=true , address=45.91.116.4:4937 , piece_id=D3TLTZCVJ7JFMSDO2UMCOBSFDS4NQP7KEBCA5YVXIXYF3VWKAUTQ, seg_pos={0 0}, upload=497173]: Long tail cancellation due to optimal threshold Dec 29 06:33:32.912899932 (567h4m32.783982967s) [noise=true , address=45.91.116.11:4120 , piece_id=YJJPC264P2Z2K7OOKVILGFRFYADYCMATJ44JJNH6XOW52T64X7FQ, seg_pos={0 0}, upload=497173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:50018->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:50018->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50018->45.91.116.11:4120: use of closed network connection Dec 29 06:33:32.912905052 (567h4m32.783988087s) [noise=true , address=45.91.116.11:4120 , piece_id=YJJPC264P2Z2K7OOKVILGFRFYADYCMATJ44JJNH6XOW52T64X7FQ, seg_pos={0 0}, upload=497173]: Long tail cancellation due to optimal threshold Dec 29 06:33:32.912920392 (567h4m32.784003427s) [seg_pos={0 0}, upload=497173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:33:32.913002728 (567h4m32.784085773s) [upload=497173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:33:32.913009250 (567h4m32.784092285s) [upload=497173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:33:33.210640011 (567h4m33.081723046s) [upload=497173]: Done uploading segments... Dec 29 06:33:33.210737957 (567h4m33.081820993s) [upload=497173]: Upload finished. err: <nil> Duration: 30.004057397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.X.....................X..................................] duration: 26.546913161s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:33:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 26.546929432s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......XX......................X..XX................X........] duration: 3.450721729s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:33:36 UTC) orbiter warning: context deadline exceeded Duration: 3.450778327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.X.....................X.................................] duration: 36.757264215s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:33:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 36.757286118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X......XX......................X..XX................X.......] duration: 3.504401833s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:33:05 UTC) orbiter warning: context deadline exceeded Duration: 3.504435869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X...........X.X..X..X........X..X.....X.X.X.X..X.X..X.....] duration: 6.0810881s trace: https://jaeger.dev.storj.io/trace/68c1c1c5f5bdba81 FAIL (2025-12-29 06:32:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 06:29:22.857214267 (567h0m22.728297312s) [upload=497115]: Starting upload Dec 29 06:29:22.857306623 (567h0m22.728389678s) [upload=497115]: Done starting upload Dec 29 06:29:22.857390302 (567h0m22.728473357s) [upload=497115]: Uploading segments... Dec 29 06:29:22.857611954 (567h0m22.728694999s) [upload=497115]: Got next segment. Inline: false Dec 29 06:29:22.857659284 (567h0m22.728742329s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:29:22.857665286 (567h0m22.728748321s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:29:23.100591629 (567h0m22.971674684s) [seg_pos={0 0}, upload=497115]: Begin upload segment... Dec 29 06:29:23.100647275 (567h0m22.97173033s) [seg_pos={0 0}, upload=497115]: Done begin upload segment. Dec 29 06:29:23.100676651 (567h0m22.971759686s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100725855 (567h0m22.97180889s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100854450 (567h0m22.971937515s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101270322 (567h0m22.972353357s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101634445 (567h0m22.97271749s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102020019 (567h0m22.973103064s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102358764 (567h0m22.973441799s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102779936 (567h0m22.973862981s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103223952 (567h0m22.974306987s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103765473 (567h0m22.974848518s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105218529 (567h0m22.976301574s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105244810 (567h0m22.976327855s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.106535557 (567h0m22.977618602s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.108938904 (567h0m22.980021939s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.110114392 (567h0m22.981197427s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.111321892 (567h0m22.982404937s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:24.801736502 (567h0m24.672819547s) [upload=497115]: Next returned nil segment Dec 29 06:29:24.801782700 (567h0m24.672865735s) [upload=497115]: Waiting for error group managing segments... Dec 29 06:29:25.085652848 (567h0m24.956735883s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.103970477 (567h0m24.975053523s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109154088 (567h0m24.980237123s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109603955 (567h0m24.98068699s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119476745 (567h0m24.99055979s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119988268 (567h0m24.991071303s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.122394750 (567h0m24.993477785s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.126521378 (567h0m24.997604414s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.136745407 (567h0m25.007828442s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.351101618 (567h0m25.222184683s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.360672432 (567h0m25.231755467s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.390064881 (567h0m25.261147916s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507207821 (567h0m25.378290856s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507228580 (567h0m25.378311615s) [seg_pos={0 0}, upload=497115]: Segment reached optimal threshold of 13 pieces. Dec 29 06:29:25.507421017 (567h0m25.378504062s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507427990 (567h0m25.378511035s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507622360 (567h0m25.378705405s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507628743 (567h0m25.378711778s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507687985 (567h0m25.37877103s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507726819 (567h0m25.378809874s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507761996 (567h0m25.378845041s) [seg_pos={0 0}, upload=497115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:29:25.507866134 (567h0m25.378949179s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:29:25.507873469 (567h0m25.378956524s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:29:25.797791722 (567h0m25.668874757s) [upload=497115]: Done uploading segments... Dec 29 06:29:25.797841928 (567h0m25.668924963s) [upload=497115]: Upload finished. err: <nil> Duration: 6.08111426s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.X.....................X................................] duration: 42.26104268s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:32:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 42.261059481s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X......XX......................X..XX................X......] duration: 4.238702399s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:32:09 UTC) orbiter warning: context deadline exceeded Duration: 4.238729031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..X...........X.X..X..X........X..X.....X.X.X.X..X.X..X....] duration: 5.671973795s trace: https://jaeger.dev.storj.io/trace/68c1c1c5f5bdba81 FAIL (2025-12-29 06:31:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 06:29:22.857214267 (567h0m22.728297312s) [upload=497115]: Starting upload Dec 29 06:29:22.857306623 (567h0m22.728389678s) [upload=497115]: Done starting upload Dec 29 06:29:22.857390302 (567h0m22.728473357s) [upload=497115]: Uploading segments... Dec 29 06:29:22.857611954 (567h0m22.728694999s) [upload=497115]: Got next segment. Inline: false Dec 29 06:29:22.857659284 (567h0m22.728742329s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:29:22.857665286 (567h0m22.728748321s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:29:23.100591629 (567h0m22.971674684s) [seg_pos={0 0}, upload=497115]: Begin upload segment... Dec 29 06:29:23.100647275 (567h0m22.97173033s) [seg_pos={0 0}, upload=497115]: Done begin upload segment. Dec 29 06:29:23.100676651 (567h0m22.971759686s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100725855 (567h0m22.97180889s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100854450 (567h0m22.971937515s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101270322 (567h0m22.972353357s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101634445 (567h0m22.97271749s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102020019 (567h0m22.973103064s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102358764 (567h0m22.973441799s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102779936 (567h0m22.973862981s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103223952 (567h0m22.974306987s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103765473 (567h0m22.974848518s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105218529 (567h0m22.976301574s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105244810 (567h0m22.976327855s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.106535557 (567h0m22.977618602s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.108938904 (567h0m22.980021939s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.110114392 (567h0m22.981197427s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.111321892 (567h0m22.982404937s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:24.801736502 (567h0m24.672819547s) [upload=497115]: Next returned nil segment Dec 29 06:29:24.801782700 (567h0m24.672865735s) [upload=497115]: Waiting for error group managing segments... Dec 29 06:29:25.085652848 (567h0m24.956735883s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.103970477 (567h0m24.975053523s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109154088 (567h0m24.980237123s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109603955 (567h0m24.98068699s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119476745 (567h0m24.99055979s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119988268 (567h0m24.991071303s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.122394750 (567h0m24.993477785s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.126521378 (567h0m24.997604414s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.136745407 (567h0m25.007828442s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.351101618 (567h0m25.222184683s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.360672432 (567h0m25.231755467s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.390064881 (567h0m25.261147916s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507207821 (567h0m25.378290856s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507228580 (567h0m25.378311615s) [seg_pos={0 0}, upload=497115]: Segment reached optimal threshold of 13 pieces. Dec 29 06:29:25.507421017 (567h0m25.378504062s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507427990 (567h0m25.378511035s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507622360 (567h0m25.378705405s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507628743 (567h0m25.378711778s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507687985 (567h0m25.37877103s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507726819 (567h0m25.378809874s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507761996 (567h0m25.378845041s) [seg_pos={0 0}, upload=497115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:29:25.507866134 (567h0m25.378949179s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:29:25.507873469 (567h0m25.378956524s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:29:25.797791722 (567h0m25.668874757s) [upload=497115]: Done uploading segments... Dec 29 06:29:25.797841928 (567h0m25.668924963s) [upload=497115]: Upload finished. err: <nil> Duration: 5.672006197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.X.....................X...............................] duration: 35.77692493s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:31:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 35.776969294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X......XX......................X..XX................X.....] duration: 3.45418278s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:31:29 UTC) orbiter warning: context deadline exceeded Duration: 3.454211454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..X...........X.X..X..X........X..X.....X.X.X.X..X.X..X...] duration: 10.882703654s trace: https://jaeger.dev.storj.io/trace/68c1c1c5f5bdba81 FAIL (2025-12-29 06:31:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 06:29:22.857214267 (567h0m22.728297312s) [upload=497115]: Starting upload Dec 29 06:29:22.857306623 (567h0m22.728389678s) [upload=497115]: Done starting upload Dec 29 06:29:22.857390302 (567h0m22.728473357s) [upload=497115]: Uploading segments... Dec 29 06:29:22.857611954 (567h0m22.728694999s) [upload=497115]: Got next segment. Inline: false Dec 29 06:29:22.857659284 (567h0m22.728742329s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:29:22.857665286 (567h0m22.728748321s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:29:23.100591629 (567h0m22.971674684s) [seg_pos={0 0}, upload=497115]: Begin upload segment... Dec 29 06:29:23.100647275 (567h0m22.97173033s) [seg_pos={0 0}, upload=497115]: Done begin upload segment. Dec 29 06:29:23.100676651 (567h0m22.971759686s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100725855 (567h0m22.97180889s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100854450 (567h0m22.971937515s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101270322 (567h0m22.972353357s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101634445 (567h0m22.97271749s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102020019 (567h0m22.973103064s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102358764 (567h0m22.973441799s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102779936 (567h0m22.973862981s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103223952 (567h0m22.974306987s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103765473 (567h0m22.974848518s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105218529 (567h0m22.976301574s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105244810 (567h0m22.976327855s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.106535557 (567h0m22.977618602s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.108938904 (567h0m22.980021939s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.110114392 (567h0m22.981197427s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.111321892 (567h0m22.982404937s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:24.801736502 (567h0m24.672819547s) [upload=497115]: Next returned nil segment Dec 29 06:29:24.801782700 (567h0m24.672865735s) [upload=497115]: Waiting for error group managing segments... Dec 29 06:29:25.085652848 (567h0m24.956735883s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.103970477 (567h0m24.975053523s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109154088 (567h0m24.980237123s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109603955 (567h0m24.98068699s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119476745 (567h0m24.99055979s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119988268 (567h0m24.991071303s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.122394750 (567h0m24.993477785s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.126521378 (567h0m24.997604414s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.136745407 (567h0m25.007828442s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.351101618 (567h0m25.222184683s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.360672432 (567h0m25.231755467s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.390064881 (567h0m25.261147916s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507207821 (567h0m25.378290856s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507228580 (567h0m25.378311615s) [seg_pos={0 0}, upload=497115]: Segment reached optimal threshold of 13 pieces. Dec 29 06:29:25.507421017 (567h0m25.378504062s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507427990 (567h0m25.378511035s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507622360 (567h0m25.378705405s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507628743 (567h0m25.378711778s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507687985 (567h0m25.37877103s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507726819 (567h0m25.378809874s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507761996 (567h0m25.378845041s) [seg_pos={0 0}, upload=497115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:29:25.507866134 (567h0m25.378949179s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:29:25.507873469 (567h0m25.378956524s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:29:25.797791722 (567h0m25.668874757s) [upload=497115]: Done uploading segments... Dec 29 06:29:25.797841928 (567h0m25.668924963s) [upload=497115]: Upload finished. err: <nil> Duration: 10.8827278s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X......XX......................X..XX................X....] duration: 4.507976023s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:31:10 UTC) orbiter warning: context deadline exceeded Duration: 4.508003747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X..X...........X.X..X..X........X..X.....X.X.X.X..X.X..X..] duration: 5.717524882s trace: https://jaeger.dev.storj.io/trace/68c1c1c5f5bdba81 FAIL (2025-12-29 06:31:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 06:29:22.857214267 (567h0m22.728297312s) [upload=497115]: Starting upload Dec 29 06:29:22.857306623 (567h0m22.728389678s) [upload=497115]: Done starting upload Dec 29 06:29:22.857390302 (567h0m22.728473357s) [upload=497115]: Uploading segments... Dec 29 06:29:22.857611954 (567h0m22.728694999s) [upload=497115]: Got next segment. Inline: false Dec 29 06:29:22.857659284 (567h0m22.728742329s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:29:22.857665286 (567h0m22.728748321s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:29:23.100591629 (567h0m22.971674684s) [seg_pos={0 0}, upload=497115]: Begin upload segment... Dec 29 06:29:23.100647275 (567h0m22.97173033s) [seg_pos={0 0}, upload=497115]: Done begin upload segment. Dec 29 06:29:23.100676651 (567h0m22.971759686s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100725855 (567h0m22.97180889s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100854450 (567h0m22.971937515s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101270322 (567h0m22.972353357s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101634445 (567h0m22.97271749s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102020019 (567h0m22.973103064s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102358764 (567h0m22.973441799s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102779936 (567h0m22.973862981s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103223952 (567h0m22.974306987s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103765473 (567h0m22.974848518s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105218529 (567h0m22.976301574s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105244810 (567h0m22.976327855s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.106535557 (567h0m22.977618602s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.108938904 (567h0m22.980021939s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.110114392 (567h0m22.981197427s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.111321892 (567h0m22.982404937s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:24.801736502 (567h0m24.672819547s) [upload=497115]: Next returned nil segment Dec 29 06:29:24.801782700 (567h0m24.672865735s) [upload=497115]: Waiting for error group managing segments... Dec 29 06:29:25.085652848 (567h0m24.956735883s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.103970477 (567h0m24.975053523s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109154088 (567h0m24.980237123s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109603955 (567h0m24.98068699s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119476745 (567h0m24.99055979s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119988268 (567h0m24.991071303s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.122394750 (567h0m24.993477785s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.126521378 (567h0m24.997604414s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.136745407 (567h0m25.007828442s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.351101618 (567h0m25.222184683s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.360672432 (567h0m25.231755467s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.390064881 (567h0m25.261147916s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507207821 (567h0m25.378290856s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507228580 (567h0m25.378311615s) [seg_pos={0 0}, upload=497115]: Segment reached optimal threshold of 13 pieces. Dec 29 06:29:25.507421017 (567h0m25.378504062s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507427990 (567h0m25.378511035s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507622360 (567h0m25.378705405s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507628743 (567h0m25.378711778s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507687985 (567h0m25.37877103s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507726819 (567h0m25.378809874s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507761996 (567h0m25.378845041s) [seg_pos={0 0}, upload=497115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:29:25.507866134 (567h0m25.378949179s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:29:25.507873469 (567h0m25.378956524s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:29:25.797791722 (567h0m25.668874757s) [upload=497115]: Done uploading segments... Dec 29 06:29:25.797841928 (567h0m25.668924963s) [upload=497115]: Upload finished. err: <nil> Duration: 5.717566182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.X.....................X..............................] duration: 15.140754699s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:30:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 15.140789114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.X.....................X.............................] duration: 35.897413685s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:30:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 35.897448542s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X......XX......................X..XX................X...] duration: 3.395376031s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:30:07 UTC) orbiter warning: context deadline exceeded Duration: 3.395405836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X..X...........X.X..X..X........X..X.....X.X.X.X..X.X..X.] duration: 5.668545188s trace: https://jaeger.dev.storj.io/trace/68c1c1c5f5bdba81 FAIL (2025-12-29 06:30:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 06:29:22.857214267 (567h0m22.728297312s) [upload=497115]: Starting upload Dec 29 06:29:22.857306623 (567h0m22.728389678s) [upload=497115]: Done starting upload Dec 29 06:29:22.857390302 (567h0m22.728473357s) [upload=497115]: Uploading segments... Dec 29 06:29:22.857611954 (567h0m22.728694999s) [upload=497115]: Got next segment. Inline: false Dec 29 06:29:22.857659284 (567h0m22.728742329s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:29:22.857665286 (567h0m22.728748321s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:29:23.100591629 (567h0m22.971674684s) [seg_pos={0 0}, upload=497115]: Begin upload segment... Dec 29 06:29:23.100647275 (567h0m22.97173033s) [seg_pos={0 0}, upload=497115]: Done begin upload segment. Dec 29 06:29:23.100676651 (567h0m22.971759686s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100725855 (567h0m22.97180889s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100854450 (567h0m22.971937515s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101270322 (567h0m22.972353357s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101634445 (567h0m22.97271749s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102020019 (567h0m22.973103064s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102358764 (567h0m22.973441799s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102779936 (567h0m22.973862981s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103223952 (567h0m22.974306987s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103765473 (567h0m22.974848518s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105218529 (567h0m22.976301574s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105244810 (567h0m22.976327855s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.106535557 (567h0m22.977618602s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.108938904 (567h0m22.980021939s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.110114392 (567h0m22.981197427s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.111321892 (567h0m22.982404937s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:24.801736502 (567h0m24.672819547s) [upload=497115]: Next returned nil segment Dec 29 06:29:24.801782700 (567h0m24.672865735s) [upload=497115]: Waiting for error group managing segments... Dec 29 06:29:25.085652848 (567h0m24.956735883s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.103970477 (567h0m24.975053523s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109154088 (567h0m24.980237123s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109603955 (567h0m24.98068699s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119476745 (567h0m24.99055979s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119988268 (567h0m24.991071303s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.122394750 (567h0m24.993477785s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.126521378 (567h0m24.997604414s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.136745407 (567h0m25.007828442s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.351101618 (567h0m25.222184683s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.360672432 (567h0m25.231755467s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.390064881 (567h0m25.261147916s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507207821 (567h0m25.378290856s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507228580 (567h0m25.378311615s) [seg_pos={0 0}, upload=497115]: Segment reached optimal threshold of 13 pieces. Dec 29 06:29:25.507421017 (567h0m25.378504062s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507427990 (567h0m25.378511035s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507622360 (567h0m25.378705405s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507628743 (567h0m25.378711778s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507687985 (567h0m25.37877103s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507726819 (567h0m25.378809874s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507761996 (567h0m25.378845041s) [seg_pos={0 0}, upload=497115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:29:25.507866134 (567h0m25.378949179s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:29:25.507873469 (567h0m25.378956524s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:29:25.797791722 (567h0m25.668874757s) [upload=497115]: Done uploading segments... Dec 29 06:29:25.797841928 (567h0m25.668924963s) [upload=497115]: Upload finished. err: <nil> Duration: 5.668574322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.X..X...........X.X..X..X........X..X.....X.X.X.X..X.X..X] duration: 30.002738293s trace: https://jaeger.dev.storj.io/trace/68c1c1c5f5bdba81 FAIL (2025-12-29 06:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 06:29:22.857214267 (567h0m22.728297312s) [upload=497115]: Starting upload Dec 29 06:29:22.857306623 (567h0m22.728389678s) [upload=497115]: Done starting upload Dec 29 06:29:22.857390302 (567h0m22.728473357s) [upload=497115]: Uploading segments... Dec 29 06:29:22.857611954 (567h0m22.728694999s) [upload=497115]: Got next segment. Inline: false Dec 29 06:29:22.857659284 (567h0m22.728742329s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:29:22.857665286 (567h0m22.728748321s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:29:23.100591629 (567h0m22.971674684s) [seg_pos={0 0}, upload=497115]: Begin upload segment... Dec 29 06:29:23.100647275 (567h0m22.97173033s) [seg_pos={0 0}, upload=497115]: Done begin upload segment. Dec 29 06:29:23.100676651 (567h0m22.971759686s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100725855 (567h0m22.97180889s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.100854450 (567h0m22.971937515s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101270322 (567h0m22.972353357s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.101634445 (567h0m22.97271749s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102020019 (567h0m22.973103064s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102358764 (567h0m22.973441799s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.102779936 (567h0m22.973862981s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103223952 (567h0m22.974306987s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.103765473 (567h0m22.974848518s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105218529 (567h0m22.976301574s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.105244810 (567h0m22.976327855s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.106535557 (567h0m22.977618602s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.108938904 (567h0m22.980021939s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.110114392 (567h0m22.981197427s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:23.111321892 (567h0m22.982404937s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Uploading piece... Dec 29 06:29:24.801736502 (567h0m24.672819547s) [upload=497115]: Next returned nil segment Dec 29 06:29:24.801782700 (567h0m24.672865735s) [upload=497115]: Waiting for error group managing segments... Dec 29 06:29:25.085652848 (567h0m24.956735883s) [noise=true , address=145.79.227.50:4374 , piece_id=WHK6NXNHGC3QEAPLMGLRDB5UYIRVLKRA2BHZMQUHXU2A4BHDOCGA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.103970477 (567h0m24.975053523s) [noise=true , address=145.79.227.2:4115 , piece_id=HGMMV2NPA5AUQQLT7ACCVVPMFNQKZUNCWM4OXWBTDLRW3A2QNNVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109154088 (567h0m24.980237123s) [noise=true , address=145.79.227.9:4367 , piece_id=HB7KFA25IS6BA774VGANTFF3XNYTXE2BTKZB7SP2YWST2YZDFIUQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.109603955 (567h0m24.98068699s) [noise=true , address=145.79.227.51:4409 , piece_id=GDGKAO4GDK5NBEUKE6URPP3KUJYKUUBIQVHNI5TY2DTKWJ6QDM5Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119476745 (567h0m24.99055979s) [noise=true , address=145.79.227.37:4566 , piece_id=GB5UH5SWH3BC763W7O4ZGCVW53COYHL272RS7OAKDFFBL4NDXJTA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.119988268 (567h0m24.991071303s) [noise=true , address=145.79.227.15:4875 , piece_id=TC54F34EMSGDEAJYLQCWC462XLP7P5IN4MQVDN7ZW6B5K3D66NVQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.122394750 (567h0m24.993477785s) [noise=true , address=145.79.227.32:4835 , piece_id=KRO7QDXQKZWMDJEPBIX77D3AYNQHZE6TRHVJ63S46OUL7HVOXKEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.126521378 (567h0m24.997604414s) [noise=true , address=145.79.227.41:4085 , piece_id=SP6DEFEXEJA5QF5MHCZ4PHLE5TROHNQIWTMQJQLHAKVRSAQG46IA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.136745407 (567h0m25.007828442s) [noise=true , address=145.79.227.38:4979 , piece_id=6DBP2CPZMITI4AU2XCT6W5QBOI2Q54AEEJKVDH6RE5GTCALX66QQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.351101618 (567h0m25.222184683s) [noise=true , address=77.90.31.10:4517 , piece_id=RC2XUPONAITBF56B6I3MP235AMSLWFXQ5E6ZZJSSILDDL7DPSXRQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.360672432 (567h0m25.231755467s) [noise=true , address=77.90.31.11:4554 , piece_id=3K5GMWPDDKC2VFDOWZQ22IF4IGAENMR4HPWQZQGYUN5ZO4C6GL7Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.390064881 (567h0m25.261147916s) [noise=true , address=45.91.116.14:4190 , piece_id=6DVZF362QUBEKFEYPPL4PK22H2ZIFCLNQQDGRHXSH774VRUIPW2Q, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507207821 (567h0m25.378290856s) [noise=true , address=45.91.116.5:4299 , piece_id=WW6N6PIDO2V6J6MZINSBL7WCMOYBYMSJUEZBP3TR2KQR4RPWGAPQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: <nil> Dec 29 06:29:25.507228580 (567h0m25.378311615s) [seg_pos={0 0}, upload=497115]: Segment reached optimal threshold of 13 pieces. Dec 29 06:29:25.507421017 (567h0m25.378504062s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507427990 (567h0m25.378511035s) [noise=true , address=45.91.116.12:4225 , piece_id=WBE3FA22Z4GLUHFYS433GJBZWPHTRH2KY5Y76MYCIK3LMEXFJVVA, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507622360 (567h0m25.378705405s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507628743 (567h0m25.378711778s) [noise=true , address=145.79.227.49:4339 , piece_id=SG3QWYNNA37J6GHJAA7AKMB5PA5VTSIZZPA43OFWOBVWTAR6WGEQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507687985 (567h0m25.37877103s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 06:29:25.507726819 (567h0m25.378809874s) [noise=true , address=77.90.31.21:4880 , piece_id=P5NH33DWUDOTF2PBG65MYFOVCLO6UAGXZWQQWRQFEHULSZGTNURQ, seg_pos={0 0}, upload=497115]: Long tail cancellation due to optimal threshold Dec 29 06:29:25.507761996 (567h0m25.378845041s) [seg_pos={0 0}, upload=497115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:29:25.507866134 (567h0m25.378949179s) [upload=497115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:29:25.507873469 (567h0m25.378956524s) [upload=497115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:29:25.797791722 (567h0m25.668874757s) [upload=497115]: Done uploading segments... Dec 29 06:29:25.797841928 (567h0m25.668924963s) [upload=497115]: Upload finished. err: <nil> Duration: 30.002821613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.X.....................X............................] duration: 41.058756987s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:29:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 41.058787767s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X......XX......................X..XX................X..] duration: 3.530881226s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:29:25 UTC) orbiter warning: context deadline exceeded Duration: 3.530913117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.X.....................X...........................] duration: 6.360043128s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:28:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 6.360111196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X.X..X...........X.X..X..X........X..X.....X.X.X.X..X.X..] duration: 6.996875066s trace: https://jaeger.dev.storj.io/trace/2188b5d9c8910226 FAIL (2025-12-29 06:28:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: LILEOP22EYNOS7D6OLJCJSK2ANYR5JZA3742LYCGQ7V2Y4K6HOSA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KOCSOMEBRVYQPCBW2SBOVYP36B7SN3MKS7AVZXLUS6TBZNSDNYVQ) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: FBSLUUWPNKU2C3QC5JDME4334DB32BPA52IJOQLK5JBFWB24UDXA) context deadline exceeded ======LOGS====== Dec 29 06:26:39.309208767 (566h57m39.180291802s) [upload=497079]: Starting upload Dec 29 06:26:39.309267128 (566h57m39.180350163s) [upload=497079]: Done starting upload Dec 29 06:26:39.309353452 (566h57m39.180436487s) [upload=497079]: Uploading segments... Dec 29 06:26:39.309438424 (566h57m39.180521459s) [upload=497079]: Got next segment. Inline: false Dec 29 06:26:39.309464133 (566h57m39.180547168s) [upload=497079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:26:39.309468872 (566h57m39.180551907s) [upload=497079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:26:39.555405147 (566h57m39.426488182s) [seg_pos={0 0}, upload=497079]: Begin upload segment... Dec 29 06:26:39.555465051 (566h57m39.426548096s) [seg_pos={0 0}, upload=497079]: Done begin upload segment. Dec 29 06:26:39.555499296 (566h57m39.426582341s) [noise=true , address=145.79.227.35:4481 , piece_id=GZLOQYZL3YJUDKHW3DVT54OBQCFATTHDPGPY3X4LWGQLXJCMUQKA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.555662117 (566h57m39.426745162s) [noise=true , address=145.79.227.2:4115 , piece_id=I5CRQQXGMTTBSHY6QGKNLFD3F2W7HTVUDHBVQXV2F4MWRFSI4G4Q, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.556241800 (566h57m39.427324835s) [noise=true , address=145.79.227.6:4259 , piece_id=AKFUQDKDYM2A3UGKKXOQZISMMDEGN52H6MS2FBIJ5I7VILTDBV2Q, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.556598440 (566h57m39.427681475s) [noise=true , address=77.90.31.12:4589 , piece_id=KOCSOMEBRVYQPCBW2SBOVYP36B7SN3MKS7AVZXLUS6TBZNSDNYVQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.557040210 (566h57m39.428123255s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FBVMVJRSGRFPXLL5UOJDXXPIBCWS56GMNI3SPN574EJ2FCR3XA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.557444370 (566h57m39.428527415s) [noise=true , address=145.79.227.45:4229 , piece_id=FBSLUUWPNKU2C3QC5JDME4334DB32BPA52IJOQLK5JBFWB24UDXA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.557938811 (566h57m39.429021856s) [noise=true , address=145.79.227.4:4186 , piece_id=C2U6OFF4UQ7R76SAKWKMTOY74YRWPJORNUHXB6DJY5RHYUYN7OYQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.558417784 (566h57m39.429500829s) [noise=true , address=45.91.116.12:4225 , piece_id=57IUVCT34PKKDY474A6KKXZOINPO7YFEPX3OWRCX3ROJYXHZNSLQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.559008889 (566h57m39.430091924s) [noise=true , address=145.79.227.16:4909 , piece_id=45VDCIZQZ2HIMS5KBHRYSHCAECHZDDRBAJ5BGG6XF4ZTPXGJ7VKA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.561262581 (566h57m39.432345626s) [noise=true , address=145.79.227.50:4374 , piece_id=VKH4Y6QO64Z3OXW7L4VQHHPHQPWMCPEQJLH5BTSNFGNMXTCG4JVQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.563362339 (566h57m39.434445384s) [noise=true , address=145.79.227.9:4367 , piece_id=R54Q3CVIXKYX6J3ULDGKI7FPCHSCVW3ABUBX3HQQQC5T4QDRLFAQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.566132664 (566h57m39.43721571s) [noise=true , address=145.79.227.32:4835 , piece_id=3HJX7SXRB6DQIL4OO5C54CON7VMRG26BRYBAAFTAVMRFIH3WHMPA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.572792275 (566h57m39.44387532s) [noise=true , address=45.91.116.7:4010 , piece_id=LILEOP22EYNOS7D6OLJCJSK2ANYR5JZA3742LYCGQ7V2Y4K6HOSA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.574094885 (566h57m39.44517793s) [noise=true , address=145.79.227.51:4409 , piece_id=MMUS2AH4YXLFGEWR2OVPJHGFOUOLXRWU6ZXFFLDVXWO2SEJHE3WA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.575694531 (566h57m39.446777586s) [noise=true , address=145.79.227.7:4293 , piece_id=6AC3H72SBNY7TYSCFIBISGJZH2TLIEZRD7H6BNDJV5C6LC63KSMA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.577125326 (566h57m39.448208361s) [noise=true , address=145.79.227.41:4087 , piece_id=VJXJE7KBLHMZVGFM4CX673XCYMO3WZLUURYPIB337VKJTR6QGDKQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:41.255310550 (566h57m41.126393595s) [upload=497079]: Next returned nil segment Dec 29 06:26:41.255332091 (566h57m41.126415136s) [upload=497079]: Waiting for error group managing segments... Dec 29 06:26:41.529382380 (566h57m41.400465415s) [noise=true , address=145.79.227.2:4115 , piece_id=I5CRQQXGMTTBSHY6QGKNLFD3F2W7HTVUDHBVQXV2F4MWRFSI4G4Q, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.544143388 (566h57m41.415226433s) [noise=true , address=145.79.227.45:4229 , piece_id=FBSLUUWPNKU2C3QC5JDME4334DB32BPA52IJOQLK5JBFWB24UDXA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.545183709 (566h57m41.416266754s) [noise=true , address=145.79.227.4:4186 , piece_id=C2U6OFF4UQ7R76SAKWKMTOY74YRWPJORNUHXB6DJY5RHYUYN7OYQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.561460011 (566h57m41.432543056s) [noise=true , address=145.79.227.50:4374 , piece_id=VKH4Y6QO64Z3OXW7L4VQHHPHQPWMCPEQJLH5BTSNFGNMXTCG4JVQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.563205686 (566h57m41.434288721s) [noise=true , address=145.79.227.6:4259 , piece_id=AKFUQDKDYM2A3UGKKXOQZISMMDEGN52H6MS2FBIJ5I7VILTDBV2Q, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.563716007 (566h57m41.434799052s) [noise=true , address=145.79.227.7:4293 , piece_id=6AC3H72SBNY7TYSCFIBISGJZH2TLIEZRD7H6BNDJV5C6LC63KSMA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.564470354 (566h57m41.435553389s) [noise=true , address=145.79.227.41:4087 , piece_id=VJXJE7KBLHMZVGFM4CX673XCYMO3WZLUURYPIB337VKJTR6QGDKQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.575210614 (566h57m41.446293649s) [noise=true , address=145.79.227.9:4367 , piece_id=R54Q3CVIXKYX6J3ULDGKI7FPCHSCVW3ABUBX3HQQQC5T4QDRLFAQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.577213499 (566h57m41.448296534s) [noise=true , address=145.79.227.51:4409 , piece_id=MMUS2AH4YXLFGEWR2OVPJHGFOUOLXRWU6ZXFFLDVXWO2SEJHE3WA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.590078485 (566h57m41.46116152s) [noise=true , address=145.79.227.32:4835 , piece_id=3HJX7SXRB6DQIL4OO5C54CON7VMRG26BRYBAAFTAVMRFIH3WHMPA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.721904565 (566h57m41.59298761s) [noise=true , address=77.90.31.12:4589 , piece_id=KOCSOMEBRVYQPCBW2SBOVYP36B7SN3MKS7AVZXLUS6TBZNSDNYVQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.875427239 (566h57m41.746510285s) [noise=true , address=45.91.116.7:4010 , piece_id=LILEOP22EYNOS7D6OLJCJSK2ANYR5JZA3742LYCGQ7V2Y4K6HOSA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.939976783 (566h57m41.811059828s) [noise=true , address=45.91.116.12:4225 , piece_id=57IUVCT34PKKDY474A6KKXZOINPO7YFEPX3OWRCX3ROJYXHZNSLQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.939995739 (566h57m41.811078784s) [seg_pos={0 0}, upload=497079]: Segment reached optimal threshold of 13 pieces. Dec 29 06:26:41.940325376 (566h57m41.811408411s) [noise=true , address=145.79.227.35:4481 , piece_id=GZLOQYZL3YJUDKHW3DVT54OBQCFATTHDPGPY3X4LWGQLXJCMUQKA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 06:26:41.940332670 (566h57m41.811415715s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FBVMVJRSGRFPXLL5UOJDXXPIBCWS56GMNI3SPN574EJ2FCR3XA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.940410598 (566h57m41.811493643s) [noise=true , address=145.79.227.35:4481 , piece_id=GZLOQYZL3YJUDKHW3DVT54OBQCFATTHDPGPY3X4LWGQLXJCMUQKA, seg_pos={0 0}, upload=497079]: Long tail cancellation due to optimal threshold Dec 29 06:26:41.940471975 (566h57m41.81155501s) [noise=true , address=145.79.227.16:4909 , piece_id=45VDCIZQZ2HIMS5KBHRYSHCAECHZDDRBAJ5BGG6XF4ZTPXGJ7VKA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Dec 29 06:26:41.940478097 (566h57m41.811561132s) [noise=true , address=145.79.227.16:4909 , piece_id=45VDCIZQZ2HIMS5KBHRYSHCAECHZDDRBAJ5BGG6XF4ZTPXGJ7VKA, seg_pos={0 0}, upload=497079]: Long tail cancellation due to optimal threshold Dec 29 06:26:41.940494729 (566h57m41.811577774s) [seg_pos={0 0}, upload=497079]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 29 06:26:41.940555665 (566h57m41.8116387s) [upload=497079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:26:41.940560023 (566h57m41.811643059s) [upload=497079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:26:42.234315243 (566h57m42.105398289s) [upload=497079]: Done uploading segments... Dec 29 06:26:42.234338448 (566h57m42.105421493s) [upload=497079]: Upload finished. err: <nil> Duration: 6.996905745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.X.....................X..........................] duration: 18.998995462s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:28:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 18.999024337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X.....X......XX......................X..XX................X.] duration: 3.817892185s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:28:14 UTC) orbiter warning: context deadline exceeded Duration: 3.817918645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.X.....................X.........................] duration: 31.542855962s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:28:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 31.542883756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.X.X..X...........X.X..X..X........X..X.....X.X.X.X..X.X.] duration: 5.643404905s trace: https://jaeger.dev.storj.io/trace/2188b5d9c8910226 FAIL (2025-12-29 06:27:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: LILEOP22EYNOS7D6OLJCJSK2ANYR5JZA3742LYCGQ7V2Y4K6HOSA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KOCSOMEBRVYQPCBW2SBOVYP36B7SN3MKS7AVZXLUS6TBZNSDNYVQ) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: FBSLUUWPNKU2C3QC5JDME4334DB32BPA52IJOQLK5JBFWB24UDXA) context deadline exceeded ======LOGS====== Dec 29 06:26:39.309208767 (566h57m39.180291802s) [upload=497079]: Starting upload Dec 29 06:26:39.309267128 (566h57m39.180350163s) [upload=497079]: Done starting upload Dec 29 06:26:39.309353452 (566h57m39.180436487s) [upload=497079]: Uploading segments... Dec 29 06:26:39.309438424 (566h57m39.180521459s) [upload=497079]: Got next segment. Inline: false Dec 29 06:26:39.309464133 (566h57m39.180547168s) [upload=497079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:26:39.309468872 (566h57m39.180551907s) [upload=497079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:26:39.555405147 (566h57m39.426488182s) [seg_pos={0 0}, upload=497079]: Begin upload segment... Dec 29 06:26:39.555465051 (566h57m39.426548096s) [seg_pos={0 0}, upload=497079]: Done begin upload segment. Dec 29 06:26:39.555499296 (566h57m39.426582341s) [noise=true , address=145.79.227.35:4481 , piece_id=GZLOQYZL3YJUDKHW3DVT54OBQCFATTHDPGPY3X4LWGQLXJCMUQKA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.555662117 (566h57m39.426745162s) [noise=true , address=145.79.227.2:4115 , piece_id=I5CRQQXGMTTBSHY6QGKNLFD3F2W7HTVUDHBVQXV2F4MWRFSI4G4Q, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.556241800 (566h57m39.427324835s) [noise=true , address=145.79.227.6:4259 , piece_id=AKFUQDKDYM2A3UGKKXOQZISMMDEGN52H6MS2FBIJ5I7VILTDBV2Q, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.556598440 (566h57m39.427681475s) [noise=true , address=77.90.31.12:4589 , piece_id=KOCSOMEBRVYQPCBW2SBOVYP36B7SN3MKS7AVZXLUS6TBZNSDNYVQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.557040210 (566h57m39.428123255s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FBVMVJRSGRFPXLL5UOJDXXPIBCWS56GMNI3SPN574EJ2FCR3XA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.557444370 (566h57m39.428527415s) [noise=true , address=145.79.227.45:4229 , piece_id=FBSLUUWPNKU2C3QC5JDME4334DB32BPA52IJOQLK5JBFWB24UDXA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.557938811 (566h57m39.429021856s) [noise=true , address=145.79.227.4:4186 , piece_id=C2U6OFF4UQ7R76SAKWKMTOY74YRWPJORNUHXB6DJY5RHYUYN7OYQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.558417784 (566h57m39.429500829s) [noise=true , address=45.91.116.12:4225 , piece_id=57IUVCT34PKKDY474A6KKXZOINPO7YFEPX3OWRCX3ROJYXHZNSLQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.559008889 (566h57m39.430091924s) [noise=true , address=145.79.227.16:4909 , piece_id=45VDCIZQZ2HIMS5KBHRYSHCAECHZDDRBAJ5BGG6XF4ZTPXGJ7VKA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.561262581 (566h57m39.432345626s) [noise=true , address=145.79.227.50:4374 , piece_id=VKH4Y6QO64Z3OXW7L4VQHHPHQPWMCPEQJLH5BTSNFGNMXTCG4JVQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.563362339 (566h57m39.434445384s) [noise=true , address=145.79.227.9:4367 , piece_id=R54Q3CVIXKYX6J3ULDGKI7FPCHSCVW3ABUBX3HQQQC5T4QDRLFAQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.566132664 (566h57m39.43721571s) [noise=true , address=145.79.227.32:4835 , piece_id=3HJX7SXRB6DQIL4OO5C54CON7VMRG26BRYBAAFTAVMRFIH3WHMPA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.572792275 (566h57m39.44387532s) [noise=true , address=45.91.116.7:4010 , piece_id=LILEOP22EYNOS7D6OLJCJSK2ANYR5JZA3742LYCGQ7V2Y4K6HOSA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.574094885 (566h57m39.44517793s) [noise=true , address=145.79.227.51:4409 , piece_id=MMUS2AH4YXLFGEWR2OVPJHGFOUOLXRWU6ZXFFLDVXWO2SEJHE3WA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.575694531 (566h57m39.446777586s) [noise=true , address=145.79.227.7:4293 , piece_id=6AC3H72SBNY7TYSCFIBISGJZH2TLIEZRD7H6BNDJV5C6LC63KSMA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.577125326 (566h57m39.448208361s) [noise=true , address=145.79.227.41:4087 , piece_id=VJXJE7KBLHMZVGFM4CX673XCYMO3WZLUURYPIB337VKJTR6QGDKQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:41.255310550 (566h57m41.126393595s) [upload=497079]: Next returned nil segment Dec 29 06:26:41.255332091 (566h57m41.126415136s) [upload=497079]: Waiting for error group managing segments... Dec 29 06:26:41.529382380 (566h57m41.400465415s) [noise=true , address=145.79.227.2:4115 , piece_id=I5CRQQXGMTTBSHY6QGKNLFD3F2W7HTVUDHBVQXV2F4MWRFSI4G4Q, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.544143388 (566h57m41.415226433s) [noise=true , address=145.79.227.45:4229 , piece_id=FBSLUUWPNKU2C3QC5JDME4334DB32BPA52IJOQLK5JBFWB24UDXA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.545183709 (566h57m41.416266754s) [noise=true , address=145.79.227.4:4186 , piece_id=C2U6OFF4UQ7R76SAKWKMTOY74YRWPJORNUHXB6DJY5RHYUYN7OYQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.561460011 (566h57m41.432543056s) [noise=true , address=145.79.227.50:4374 , piece_id=VKH4Y6QO64Z3OXW7L4VQHHPHQPWMCPEQJLH5BTSNFGNMXTCG4JVQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.563205686 (566h57m41.434288721s) [noise=true , address=145.79.227.6:4259 , piece_id=AKFUQDKDYM2A3UGKKXOQZISMMDEGN52H6MS2FBIJ5I7VILTDBV2Q, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.563716007 (566h57m41.434799052s) [noise=true , address=145.79.227.7:4293 , piece_id=6AC3H72SBNY7TYSCFIBISGJZH2TLIEZRD7H6BNDJV5C6LC63KSMA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.564470354 (566h57m41.435553389s) [noise=true , address=145.79.227.41:4087 , piece_id=VJXJE7KBLHMZVGFM4CX673XCYMO3WZLUURYPIB337VKJTR6QGDKQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.575210614 (566h57m41.446293649s) [noise=true , address=145.79.227.9:4367 , piece_id=R54Q3CVIXKYX6J3ULDGKI7FPCHSCVW3ABUBX3HQQQC5T4QDRLFAQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.577213499 (566h57m41.448296534s) [noise=true , address=145.79.227.51:4409 , piece_id=MMUS2AH4YXLFGEWR2OVPJHGFOUOLXRWU6ZXFFLDVXWO2SEJHE3WA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.590078485 (566h57m41.46116152s) [noise=true , address=145.79.227.32:4835 , piece_id=3HJX7SXRB6DQIL4OO5C54CON7VMRG26BRYBAAFTAVMRFIH3WHMPA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.721904565 (566h57m41.59298761s) [noise=true , address=77.90.31.12:4589 , piece_id=KOCSOMEBRVYQPCBW2SBOVYP36B7SN3MKS7AVZXLUS6TBZNSDNYVQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.875427239 (566h57m41.746510285s) [noise=true , address=45.91.116.7:4010 , piece_id=LILEOP22EYNOS7D6OLJCJSK2ANYR5JZA3742LYCGQ7V2Y4K6HOSA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.939976783 (566h57m41.811059828s) [noise=true , address=45.91.116.12:4225 , piece_id=57IUVCT34PKKDY474A6KKXZOINPO7YFEPX3OWRCX3ROJYXHZNSLQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.939995739 (566h57m41.811078784s) [seg_pos={0 0}, upload=497079]: Segment reached optimal threshold of 13 pieces. Dec 29 06:26:41.940325376 (566h57m41.811408411s) [noise=true , address=145.79.227.35:4481 , piece_id=GZLOQYZL3YJUDKHW3DVT54OBQCFATTHDPGPY3X4LWGQLXJCMUQKA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 06:26:41.940332670 (566h57m41.811415715s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FBVMVJRSGRFPXLL5UOJDXXPIBCWS56GMNI3SPN574EJ2FCR3XA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.940410598 (566h57m41.811493643s) [noise=true , address=145.79.227.35:4481 , piece_id=GZLOQYZL3YJUDKHW3DVT54OBQCFATTHDPGPY3X4LWGQLXJCMUQKA, seg_pos={0 0}, upload=497079]: Long tail cancellation due to optimal threshold Dec 29 06:26:41.940471975 (566h57m41.81155501s) [noise=true , address=145.79.227.16:4909 , piece_id=45VDCIZQZ2HIMS5KBHRYSHCAECHZDDRBAJ5BGG6XF4ZTPXGJ7VKA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Dec 29 06:26:41.940478097 (566h57m41.811561132s) [noise=true , address=145.79.227.16:4909 , piece_id=45VDCIZQZ2HIMS5KBHRYSHCAECHZDDRBAJ5BGG6XF4ZTPXGJ7VKA, seg_pos={0 0}, upload=497079]: Long tail cancellation due to optimal threshold Dec 29 06:26:41.940494729 (566h57m41.811577774s) [seg_pos={0 0}, upload=497079]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 29 06:26:41.940555665 (566h57m41.8116387s) [upload=497079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:26:41.940560023 (566h57m41.811643059s) [upload=497079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:26:42.234315243 (566h57m42.105398289s) [upload=497079]: Done uploading segments... Dec 29 06:26:42.234338448 (566h57m42.105421493s) [upload=497079]: Upload finished. err: <nil> Duration: 5.643431665s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X.....X......XX......................X..XX................X] duration: 5.000755211s trace: https://jaeger.dev.storj.io/trace/73575846c6215124 FAIL (2025-12-29 06:27:25 UTC) orbiter warning: context deadline exceeded Duration: 5.000777804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.X.....................X........................] duration: 44.04376938s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:27:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 44.043794618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.X.X..X...........X.X..X..X........X..X.....X.X.X.X..X.X] duration: 30.00261004s trace: https://jaeger.dev.storj.io/trace/2188b5d9c8910226 FAIL (2025-12-29 06:27:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: LILEOP22EYNOS7D6OLJCJSK2ANYR5JZA3742LYCGQ7V2Y4K6HOSA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KOCSOMEBRVYQPCBW2SBOVYP36B7SN3MKS7AVZXLUS6TBZNSDNYVQ) context deadline exceeded; piecestore: (Node ID: 12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, Piece ID: FBSLUUWPNKU2C3QC5JDME4334DB32BPA52IJOQLK5JBFWB24UDXA) context deadline exceeded ======LOGS====== Dec 29 06:26:39.309208767 (566h57m39.180291802s) [upload=497079]: Starting upload Dec 29 06:26:39.309267128 (566h57m39.180350163s) [upload=497079]: Done starting upload Dec 29 06:26:39.309353452 (566h57m39.180436487s) [upload=497079]: Uploading segments... Dec 29 06:26:39.309438424 (566h57m39.180521459s) [upload=497079]: Got next segment. Inline: false Dec 29 06:26:39.309464133 (566h57m39.180547168s) [upload=497079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:26:39.309468872 (566h57m39.180551907s) [upload=497079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:26:39.555405147 (566h57m39.426488182s) [seg_pos={0 0}, upload=497079]: Begin upload segment... Dec 29 06:26:39.555465051 (566h57m39.426548096s) [seg_pos={0 0}, upload=497079]: Done begin upload segment. Dec 29 06:26:39.555499296 (566h57m39.426582341s) [noise=true , address=145.79.227.35:4481 , piece_id=GZLOQYZL3YJUDKHW3DVT54OBQCFATTHDPGPY3X4LWGQLXJCMUQKA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.555662117 (566h57m39.426745162s) [noise=true , address=145.79.227.2:4115 , piece_id=I5CRQQXGMTTBSHY6QGKNLFD3F2W7HTVUDHBVQXV2F4MWRFSI4G4Q, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.556241800 (566h57m39.427324835s) [noise=true , address=145.79.227.6:4259 , piece_id=AKFUQDKDYM2A3UGKKXOQZISMMDEGN52H6MS2FBIJ5I7VILTDBV2Q, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.556598440 (566h57m39.427681475s) [noise=true , address=77.90.31.12:4589 , piece_id=KOCSOMEBRVYQPCBW2SBOVYP36B7SN3MKS7AVZXLUS6TBZNSDNYVQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.557040210 (566h57m39.428123255s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FBVMVJRSGRFPXLL5UOJDXXPIBCWS56GMNI3SPN574EJ2FCR3XA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.557444370 (566h57m39.428527415s) [noise=true , address=145.79.227.45:4229 , piece_id=FBSLUUWPNKU2C3QC5JDME4334DB32BPA52IJOQLK5JBFWB24UDXA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.557938811 (566h57m39.429021856s) [noise=true , address=145.79.227.4:4186 , piece_id=C2U6OFF4UQ7R76SAKWKMTOY74YRWPJORNUHXB6DJY5RHYUYN7OYQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.558417784 (566h57m39.429500829s) [noise=true , address=45.91.116.12:4225 , piece_id=57IUVCT34PKKDY474A6KKXZOINPO7YFEPX3OWRCX3ROJYXHZNSLQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.559008889 (566h57m39.430091924s) [noise=true , address=145.79.227.16:4909 , piece_id=45VDCIZQZ2HIMS5KBHRYSHCAECHZDDRBAJ5BGG6XF4ZTPXGJ7VKA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.561262581 (566h57m39.432345626s) [noise=true , address=145.79.227.50:4374 , piece_id=VKH4Y6QO64Z3OXW7L4VQHHPHQPWMCPEQJLH5BTSNFGNMXTCG4JVQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.563362339 (566h57m39.434445384s) [noise=true , address=145.79.227.9:4367 , piece_id=R54Q3CVIXKYX6J3ULDGKI7FPCHSCVW3ABUBX3HQQQC5T4QDRLFAQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.566132664 (566h57m39.43721571s) [noise=true , address=145.79.227.32:4835 , piece_id=3HJX7SXRB6DQIL4OO5C54CON7VMRG26BRYBAAFTAVMRFIH3WHMPA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.572792275 (566h57m39.44387532s) [noise=true , address=45.91.116.7:4010 , piece_id=LILEOP22EYNOS7D6OLJCJSK2ANYR5JZA3742LYCGQ7V2Y4K6HOSA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.574094885 (566h57m39.44517793s) [noise=true , address=145.79.227.51:4409 , piece_id=MMUS2AH4YXLFGEWR2OVPJHGFOUOLXRWU6ZXFFLDVXWO2SEJHE3WA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.575694531 (566h57m39.446777586s) [noise=true , address=145.79.227.7:4293 , piece_id=6AC3H72SBNY7TYSCFIBISGJZH2TLIEZRD7H6BNDJV5C6LC63KSMA, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:39.577125326 (566h57m39.448208361s) [noise=true , address=145.79.227.41:4087 , piece_id=VJXJE7KBLHMZVGFM4CX673XCYMO3WZLUURYPIB337VKJTR6QGDKQ, seg_pos={0 0}, upload=497079]: Uploading piece... Dec 29 06:26:41.255310550 (566h57m41.126393595s) [upload=497079]: Next returned nil segment Dec 29 06:26:41.255332091 (566h57m41.126415136s) [upload=497079]: Waiting for error group managing segments... Dec 29 06:26:41.529382380 (566h57m41.400465415s) [noise=true , address=145.79.227.2:4115 , piece_id=I5CRQQXGMTTBSHY6QGKNLFD3F2W7HTVUDHBVQXV2F4MWRFSI4G4Q, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.544143388 (566h57m41.415226433s) [noise=true , address=145.79.227.45:4229 , piece_id=FBSLUUWPNKU2C3QC5JDME4334DB32BPA52IJOQLK5JBFWB24UDXA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.545183709 (566h57m41.416266754s) [noise=true , address=145.79.227.4:4186 , piece_id=C2U6OFF4UQ7R76SAKWKMTOY74YRWPJORNUHXB6DJY5RHYUYN7OYQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.561460011 (566h57m41.432543056s) [noise=true , address=145.79.227.50:4374 , piece_id=VKH4Y6QO64Z3OXW7L4VQHHPHQPWMCPEQJLH5BTSNFGNMXTCG4JVQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.563205686 (566h57m41.434288721s) [noise=true , address=145.79.227.6:4259 , piece_id=AKFUQDKDYM2A3UGKKXOQZISMMDEGN52H6MS2FBIJ5I7VILTDBV2Q, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.563716007 (566h57m41.434799052s) [noise=true , address=145.79.227.7:4293 , piece_id=6AC3H72SBNY7TYSCFIBISGJZH2TLIEZRD7H6BNDJV5C6LC63KSMA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.564470354 (566h57m41.435553389s) [noise=true , address=145.79.227.41:4087 , piece_id=VJXJE7KBLHMZVGFM4CX673XCYMO3WZLUURYPIB337VKJTR6QGDKQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.575210614 (566h57m41.446293649s) [noise=true , address=145.79.227.9:4367 , piece_id=R54Q3CVIXKYX6J3ULDGKI7FPCHSCVW3ABUBX3HQQQC5T4QDRLFAQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.577213499 (566h57m41.448296534s) [noise=true , address=145.79.227.51:4409 , piece_id=MMUS2AH4YXLFGEWR2OVPJHGFOUOLXRWU6ZXFFLDVXWO2SEJHE3WA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.590078485 (566h57m41.46116152s) [noise=true , address=145.79.227.32:4835 , piece_id=3HJX7SXRB6DQIL4OO5C54CON7VMRG26BRYBAAFTAVMRFIH3WHMPA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.721904565 (566h57m41.59298761s) [noise=true , address=77.90.31.12:4589 , piece_id=KOCSOMEBRVYQPCBW2SBOVYP36B7SN3MKS7AVZXLUS6TBZNSDNYVQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.875427239 (566h57m41.746510285s) [noise=true , address=45.91.116.7:4010 , piece_id=LILEOP22EYNOS7D6OLJCJSK2ANYR5JZA3742LYCGQ7V2Y4K6HOSA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.939976783 (566h57m41.811059828s) [noise=true , address=45.91.116.12:4225 , piece_id=57IUVCT34PKKDY474A6KKXZOINPO7YFEPX3OWRCX3ROJYXHZNSLQ, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.939995739 (566h57m41.811078784s) [seg_pos={0 0}, upload=497079]: Segment reached optimal threshold of 13 pieces. Dec 29 06:26:41.940325376 (566h57m41.811408411s) [noise=true , address=145.79.227.35:4481 , piece_id=GZLOQYZL3YJUDKHW3DVT54OBQCFATTHDPGPY3X4LWGQLXJCMUQKA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 06:26:41.940332670 (566h57m41.811415715s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FBVMVJRSGRFPXLL5UOJDXXPIBCWS56GMNI3SPN574EJ2FCR3XA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: <nil> Dec 29 06:26:41.940410598 (566h57m41.811493643s) [noise=true , address=145.79.227.35:4481 , piece_id=GZLOQYZL3YJUDKHW3DVT54OBQCFATTHDPGPY3X4LWGQLXJCMUQKA, seg_pos={0 0}, upload=497079]: Long tail cancellation due to optimal threshold Dec 29 06:26:41.940471975 (566h57m41.81155501s) [noise=true , address=145.79.227.16:4909 , piece_id=45VDCIZQZ2HIMS5KBHRYSHCAECHZDDRBAJ5BGG6XF4ZTPXGJ7VKA, seg_pos={0 0}, upload=497079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Dec 29 06:26:41.940478097 (566h57m41.811561132s) [noise=true , address=145.79.227.16:4909 , piece_id=45VDCIZQZ2HIMS5KBHRYSHCAECHZDDRBAJ5BGG6XF4ZTPXGJ7VKA, seg_pos={0 0}, upload=497079]: Long tail cancellation due to optimal threshold Dec 29 06:26:41.940494729 (566h57m41.811577774s) [seg_pos={0 0}, upload=497079]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 29 06:26:41.940555665 (566h57m41.8116387s) [upload=497079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:26:41.940560023 (566h57m41.811643059s) [upload=497079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:26:42.234315243 (566h57m42.105398289s) [upload=497079]: Done uploading segments... Dec 29 06:26:42.234338448 (566h57m42.105421493s) [upload=497079]: Upload finished. err: <nil> Duration: 30.002633074s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X.X.....X......XX......................X..XX................] duration: 3.739612797s trace: https://jaeger.dev.storj.io/trace/20c51758eaefede7 FAIL (2025-12-29 06:26:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.739637786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X.X.X..X...........X.X..X..X........X..X.....X.X.X.X..X.] duration: 10.325792729s trace: https://jaeger.dev.storj.io/trace/6a78ec4afb737e96 FAIL (2025-12-29 06:26:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:05.795586558 (566h56m5.666669603s) [upload=497056]: Starting upload Dec 29 06:25:05.795709763 (566h56m5.666792818s) [upload=497056]: Done starting upload Dec 29 06:25:05.795778093 (566h56m5.666861138s) [upload=497056]: Uploading segments... Dec 29 06:25:05.796017770 (566h56m5.667100815s) [upload=497056]: Got next segment. Inline: false Dec 29 06:25:05.796092722 (566h56m5.667175758s) [upload=497056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:25:05.796099856 (566h56m5.667182901s) [upload=497056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:25:06.044839869 (566h56m5.915922934s) [seg_pos={0 0}, upload=497056]: Begin upload segment... Dec 29 06:25:06.044902208 (566h56m5.915985253s) [seg_pos={0 0}, upload=497056]: Done begin upload segment. Dec 29 06:25:06.045011717 (566h56m5.916094762s) [noise=true , address=145.79.227.13:4803 , piece_id=XMIAZZNFWF3HIV4E37YRPURDUIRQ2EKYUMA3U4Q3SYTXII7M33RQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.045019912 (566h56m5.916102957s) [noise=true , address=145.79.227.44:4193 , piece_id=AFO4EOIFF6HTIHR7GDODVYAT3FDSZLCJKBFQY6VTOVXLERYTX33Q, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.045772576 (566h56m5.916855621s) [noise=true , address=45.91.116.10:4081 , piece_id=DCEKNQFTUYV3LZN2UCT64DW7HC6IXRVERCA4JHFGKLEI7XEG2G5Q, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.047073172 (566h56m5.918156217s) [noise=true , address=145.79.227.52:4445 , piece_id=QZBYJYMOL2SJSOGTJITRNQXM74XAFQLGROMB46KBCOLGVODBZSDQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.047210483 (566h56m5.918293528s) [noise=true , address=145.79.227.6:4257 , piece_id=4HSGSZCPR575P2LRMQIHRWXEPSFE3ICB5Q4GBVZFV555EZQA6IEQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.047696980 (566h56m5.918780025s) [noise=true , address=45.91.116.6:4976 , piece_id=BACAIDU6JILQK26XFCCWTR2X3RNSNLTLCLWTUEW2W5Q4KP42EE3Q, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.047923000 (566h56m5.919006045s) [noise=true , address=45.91.116.13:4154 , piece_id=TEC2A2BYK6CLATG6ASPARLL6POUJZO3NRXCSA5BUCV7YGLPO5T4A, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.048239704 (566h56m5.919322749s) [noise=true , address=45.91.116.9:4264 , piece_id=JKYFJA6RFQYC4GUMRBWMHEUPVDFFJJMZBODM3TJXEJ2RRIMVE47A, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.048678108 (566h56m5.919761153s) [noise=true , address=45.91.116.7:4011 , piece_id=NKFARSCWTX4EJYIHNEWPVGBFRWU4ELCOCA2QP2FPDFNAUS5ZBGGA, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.048765144 (566h56m5.919848189s) [noise=true , address=145.79.227.45:4230 , piece_id=2LOJX6NXV5KC3G5NGRVHGHG24EREPP3HXZYVCQMSYJO4MZESHIHQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.049280024 (566h56m5.920363069s) [noise=true , address=77.90.31.21:4877 , piece_id=JG3DAF6H2JL2WZQW7BVFGOAUXQL4U4BKL6PKCBKXBUR2DXWNAW4A, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.049312225 (566h56m5.92039527s) [noise=true , address=145.79.227.46:4602 , piece_id=7NMXTJZFVSO4OFJG4OUJETTLOTP2OYCZEILOB25MN5KS6DIAKXCQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.049942565 (566h56m5.92102561s) [noise=true , address=77.90.31.13:4626 , piece_id=6IGEYD3T5DUYPKHTEODXTLZ3ABFJCMQA5GIEQJ2D73UJXPZAUJLQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.050035573 (566h56m5.921118618s) [noise=true , address=145.79.227.3:4151 , piece_id=7SJGKREYWLKB7F2OVG3DUVZ5FUHD4DYCMRUNS42EFSIE3LY75WWA, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.050333991 (566h56m5.921417036s) [noise=true , address=145.79.227.14:4837 , piece_id=UL7WVMCG53MB6OAYZD2LBQQ5G73KVK7L34C3IT2GVUQNK2V5S6UA, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.051169282 (566h56m5.922252317s) [noise=true , address=145.79.227.37:4566 , piece_id=K6VKS2VOSRGBEOEH7ZLFLIKI7H4YZUUIAYF3XR7UWQEOTEPQ3JJA, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:07.733381945 (566h56m7.60446498s) [upload=497056]: Next returned nil segment Dec 29 06:25:07.733409517 (566h56m7.604492562s) [upload=497056]: Waiting for error group managing segments... Dec 29 06:25:08.014977571 (566h56m7.886060606s) [noise=true , address=145.79.227.6:4257 , piece_id=4HSGSZCPR575P2LRMQIHRWXEPSFE3ICB5Q4GBVZFV555EZQA6IEQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.015241253 (566h56m7.886324288s) [noise=true , address=145.79.227.13:4803 , piece_id=XMIAZZNFWF3HIV4E37YRPURDUIRQ2EKYUMA3U4Q3SYTXII7M33RQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.031746432 (566h56m7.902829477s) [noise=true , address=145.79.227.45:4230 , piece_id=2LOJX6NXV5KC3G5NGRVHGHG24EREPP3HXZYVCQMSYJO4MZESHIHQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.032307901 (566h56m7.903390946s) [noise=true , address=145.79.227.37:4566 , piece_id=K6VKS2VOSRGBEOEH7ZLFLIKI7H4YZUUIAYF3XR7UWQEOTEPQ3JJA, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.045895845 (566h56m7.916978891s) [noise=true , address=145.79.227.52:4445 , piece_id=QZBYJYMOL2SJSOGTJITRNQXM74XAFQLGROMB46KBCOLGVODBZSDQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.048228457 (566h56m7.919311492s) [noise=true , address=145.79.227.14:4837 , piece_id=UL7WVMCG53MB6OAYZD2LBQQ5G73KVK7L34C3IT2GVUQNK2V5S6UA, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.062350607 (566h56m7.933433653s) [noise=true , address=145.79.227.3:4151 , piece_id=7SJGKREYWLKB7F2OVG3DUVZ5FUHD4DYCMRUNS42EFSIE3LY75WWA, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.065195234 (566h56m7.93627829s) [noise=true , address=145.79.227.46:4602 , piece_id=7NMXTJZFVSO4OFJG4OUJETTLOTP2OYCZEILOB25MN5KS6DIAKXCQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.340395604 (566h56m8.211478659s) [noise=true , address=45.91.116.7:4011 , piece_id=NKFARSCWTX4EJYIHNEWPVGBFRWU4ELCOCA2QP2FPDFNAUS5ZBGGA, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.423699278 (566h56m8.294782313s) [noise=true , address=45.91.116.10:4081 , piece_id=DCEKNQFTUYV3LZN2UCT64DW7HC6IXRVERCA4JHFGKLEI7XEG2G5Q, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.441457303 (566h56m8.312540348s) [noise=true , address=45.91.116.9:4264 , piece_id=JKYFJA6RFQYC4GUMRBWMHEUPVDFFJJMZBODM3TJXEJ2RRIMVE47A, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.448760900 (566h56m8.319843945s) [noise=true , address=77.90.31.13:4626 , piece_id=6IGEYD3T5DUYPKHTEODXTLZ3ABFJCMQA5GIEQJ2D73UJXPZAUJLQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.471800380 (566h56m8.342883425s) [noise=true , address=45.91.116.6:4976 , piece_id=BACAIDU6JILQK26XFCCWTR2X3RNSNLTLCLWTUEW2W5Q4KP42EE3Q, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.471826330 (566h56m8.342909375s) [seg_pos={0 0}, upload=497056]: Segment reached optimal threshold of 13 pieces. Dec 29 06:25:08.472055657 (566h56m8.343138722s) [noise=true , address=77.90.31.21:4877 , piece_id=JG3DAF6H2JL2WZQW7BVFGOAUXQL4U4BKL6PKCBKXBUR2DXWNAW4A, seg_pos={0 0}, upload=497056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:51866->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:51866->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51866->77.90.31.21:4877: use of closed network connection Dec 29 06:25:08.472065856 (566h56m8.343148891s) [noise=true , address=77.90.31.21:4877 , piece_id=JG3DAF6H2JL2WZQW7BVFGOAUXQL4U4BKL6PKCBKXBUR2DXWNAW4A, seg_pos={0 0}, upload=497056]: Long tail cancellation due to optimal threshold Dec 29 06:25:08.472166458 (566h56m8.343249523s) [noise=true , address=145.79.227.44:4193 , piece_id=AFO4EOIFF6HTIHR7GDODVYAT3FDSZLCJKBFQY6VTOVXLERYTX33Q, seg_pos={0 0}, upload=497056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 06:25:08.472176657 (566h56m8.343259702s) [noise=true , address=145.79.227.44:4193 , piece_id=AFO4EOIFF6HTIHR7GDODVYAT3FDSZLCJKBFQY6VTOVXLERYTX33Q, seg_pos={0 0}, upload=497056]: Long tail cancellation due to optimal threshold Dec 29 06:25:08.472404331 (566h56m8.343487366s) [noise=true , address=45.91.116.13:4154 , piece_id=TEC2A2BYK6CLATG6ASPARLL6POUJZO3NRXCSA5BUCV7YGLPO5T4A, seg_pos={0 0}, upload=497056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:49840->45.91.116.13:4154: use of closed network connection; context canceled Dec 29 06:25:08.472414480 (566h56m8.343497525s) [noise=true , address=45.91.116.13:4154 , piece_id=TEC2A2BYK6CLATG6ASPARLL6POUJZO3NRXCSA5BUCV7YGLPO5T4A, seg_pos={0 0}, upload=497056]: Long tail cancellation due to optimal threshold Dec 29 06:25:08.472438226 (566h56m8.343521271s) [seg_pos={0 0}, upload=497056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:25:08.472577671 (566h56m8.343660716s) [upload=497056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:25:08.472598741 (566h56m8.343681796s) [upload=497056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:25:08.772903293 (566h56m8.643986338s) [upload=497056]: Done uploading segments... Dec 29 06:25:08.772924694 (566h56m8.644007739s) [upload=497056]: Upload finished. err: <nil> Duration: 10.325817978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.X.....................X.......................] duration: 33.159513068s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:26:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 33.159538078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X....X.X.X..X...........X.X..X..X........X..X.....X.X.X.X..X] duration: 30.002387086s trace: https://jaeger.dev.storj.io/trace/6a78ec4afb737e96 FAIL (2025-12-29 06:25:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:05.795586558 (566h56m5.666669603s) [upload=497056]: Starting upload Dec 29 06:25:05.795709763 (566h56m5.666792818s) [upload=497056]: Done starting upload Dec 29 06:25:05.795778093 (566h56m5.666861138s) [upload=497056]: Uploading segments... Dec 29 06:25:05.796017770 (566h56m5.667100815s) [upload=497056]: Got next segment. Inline: false Dec 29 06:25:05.796092722 (566h56m5.667175758s) [upload=497056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:25:05.796099856 (566h56m5.667182901s) [upload=497056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:25:06.044839869 (566h56m5.915922934s) [seg_pos={0 0}, upload=497056]: Begin upload segment... Dec 29 06:25:06.044902208 (566h56m5.915985253s) [seg_pos={0 0}, upload=497056]: Done begin upload segment. Dec 29 06:25:06.045011717 (566h56m5.916094762s) [noise=true , address=145.79.227.13:4803 , piece_id=XMIAZZNFWF3HIV4E37YRPURDUIRQ2EKYUMA3U4Q3SYTXII7M33RQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.045019912 (566h56m5.916102957s) [noise=true , address=145.79.227.44:4193 , piece_id=AFO4EOIFF6HTIHR7GDODVYAT3FDSZLCJKBFQY6VTOVXLERYTX33Q, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.045772576 (566h56m5.916855621s) [noise=true , address=45.91.116.10:4081 , piece_id=DCEKNQFTUYV3LZN2UCT64DW7HC6IXRVERCA4JHFGKLEI7XEG2G5Q, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.047073172 (566h56m5.918156217s) [noise=true , address=145.79.227.52:4445 , piece_id=QZBYJYMOL2SJSOGTJITRNQXM74XAFQLGROMB46KBCOLGVODBZSDQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.047210483 (566h56m5.918293528s) [noise=true , address=145.79.227.6:4257 , piece_id=4HSGSZCPR575P2LRMQIHRWXEPSFE3ICB5Q4GBVZFV555EZQA6IEQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.047696980 (566h56m5.918780025s) [noise=true , address=45.91.116.6:4976 , piece_id=BACAIDU6JILQK26XFCCWTR2X3RNSNLTLCLWTUEW2W5Q4KP42EE3Q, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.047923000 (566h56m5.919006045s) [noise=true , address=45.91.116.13:4154 , piece_id=TEC2A2BYK6CLATG6ASPARLL6POUJZO3NRXCSA5BUCV7YGLPO5T4A, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.048239704 (566h56m5.919322749s) [noise=true , address=45.91.116.9:4264 , piece_id=JKYFJA6RFQYC4GUMRBWMHEUPVDFFJJMZBODM3TJXEJ2RRIMVE47A, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.048678108 (566h56m5.919761153s) [noise=true , address=45.91.116.7:4011 , piece_id=NKFARSCWTX4EJYIHNEWPVGBFRWU4ELCOCA2QP2FPDFNAUS5ZBGGA, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.048765144 (566h56m5.919848189s) [noise=true , address=145.79.227.45:4230 , piece_id=2LOJX6NXV5KC3G5NGRVHGHG24EREPP3HXZYVCQMSYJO4MZESHIHQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.049280024 (566h56m5.920363069s) [noise=true , address=77.90.31.21:4877 , piece_id=JG3DAF6H2JL2WZQW7BVFGOAUXQL4U4BKL6PKCBKXBUR2DXWNAW4A, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.049312225 (566h56m5.92039527s) [noise=true , address=145.79.227.46:4602 , piece_id=7NMXTJZFVSO4OFJG4OUJETTLOTP2OYCZEILOB25MN5KS6DIAKXCQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.049942565 (566h56m5.92102561s) [noise=true , address=77.90.31.13:4626 , piece_id=6IGEYD3T5DUYPKHTEODXTLZ3ABFJCMQA5GIEQJ2D73UJXPZAUJLQ, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.050035573 (566h56m5.921118618s) [noise=true , address=145.79.227.3:4151 , piece_id=7SJGKREYWLKB7F2OVG3DUVZ5FUHD4DYCMRUNS42EFSIE3LY75WWA, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.050333991 (566h56m5.921417036s) [noise=true , address=145.79.227.14:4837 , piece_id=UL7WVMCG53MB6OAYZD2LBQQ5G73KVK7L34C3IT2GVUQNK2V5S6UA, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:06.051169282 (566h56m5.922252317s) [noise=true , address=145.79.227.37:4566 , piece_id=K6VKS2VOSRGBEOEH7ZLFLIKI7H4YZUUIAYF3XR7UWQEOTEPQ3JJA, seg_pos={0 0}, upload=497056]: Uploading piece... Dec 29 06:25:07.733381945 (566h56m7.60446498s) [upload=497056]: Next returned nil segment Dec 29 06:25:07.733409517 (566h56m7.604492562s) [upload=497056]: Waiting for error group managing segments... Dec 29 06:25:08.014977571 (566h56m7.886060606s) [noise=true , address=145.79.227.6:4257 , piece_id=4HSGSZCPR575P2LRMQIHRWXEPSFE3ICB5Q4GBVZFV555EZQA6IEQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.015241253 (566h56m7.886324288s) [noise=true , address=145.79.227.13:4803 , piece_id=XMIAZZNFWF3HIV4E37YRPURDUIRQ2EKYUMA3U4Q3SYTXII7M33RQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.031746432 (566h56m7.902829477s) [noise=true , address=145.79.227.45:4230 , piece_id=2LOJX6NXV5KC3G5NGRVHGHG24EREPP3HXZYVCQMSYJO4MZESHIHQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.032307901 (566h56m7.903390946s) [noise=true , address=145.79.227.37:4566 , piece_id=K6VKS2VOSRGBEOEH7ZLFLIKI7H4YZUUIAYF3XR7UWQEOTEPQ3JJA, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.045895845 (566h56m7.916978891s) [noise=true , address=145.79.227.52:4445 , piece_id=QZBYJYMOL2SJSOGTJITRNQXM74XAFQLGROMB46KBCOLGVODBZSDQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.048228457 (566h56m7.919311492s) [noise=true , address=145.79.227.14:4837 , piece_id=UL7WVMCG53MB6OAYZD2LBQQ5G73KVK7L34C3IT2GVUQNK2V5S6UA, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.062350607 (566h56m7.933433653s) [noise=true , address=145.79.227.3:4151 , piece_id=7SJGKREYWLKB7F2OVG3DUVZ5FUHD4DYCMRUNS42EFSIE3LY75WWA, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.065195234 (566h56m7.93627829s) [noise=true , address=145.79.227.46:4602 , piece_id=7NMXTJZFVSO4OFJG4OUJETTLOTP2OYCZEILOB25MN5KS6DIAKXCQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.340395604 (566h56m8.211478659s) [noise=true , address=45.91.116.7:4011 , piece_id=NKFARSCWTX4EJYIHNEWPVGBFRWU4ELCOCA2QP2FPDFNAUS5ZBGGA, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.423699278 (566h56m8.294782313s) [noise=true , address=45.91.116.10:4081 , piece_id=DCEKNQFTUYV3LZN2UCT64DW7HC6IXRVERCA4JHFGKLEI7XEG2G5Q, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.441457303 (566h56m8.312540348s) [noise=true , address=45.91.116.9:4264 , piece_id=JKYFJA6RFQYC4GUMRBWMHEUPVDFFJJMZBODM3TJXEJ2RRIMVE47A, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.448760900 (566h56m8.319843945s) [noise=true , address=77.90.31.13:4626 , piece_id=6IGEYD3T5DUYPKHTEODXTLZ3ABFJCMQA5GIEQJ2D73UJXPZAUJLQ, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.471800380 (566h56m8.342883425s) [noise=true , address=45.91.116.6:4976 , piece_id=BACAIDU6JILQK26XFCCWTR2X3RNSNLTLCLWTUEW2W5Q4KP42EE3Q, seg_pos={0 0}, upload=497056]: Done uploading piece. err: <nil> Dec 29 06:25:08.471826330 (566h56m8.342909375s) [seg_pos={0 0}, upload=497056]: Segment reached optimal threshold of 13 pieces. Dec 29 06:25:08.472055657 (566h56m8.343138722s) [noise=true , address=77.90.31.21:4877 , piece_id=JG3DAF6H2JL2WZQW7BVFGOAUXQL4U4BKL6PKCBKXBUR2DXWNAW4A, seg_pos={0 0}, upload=497056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:51866->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:51866->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51866->77.90.31.21:4877: use of closed network connection Dec 29 06:25:08.472065856 (566h56m8.343148891s) [noise=true , address=77.90.31.21:4877 , piece_id=JG3DAF6H2JL2WZQW7BVFGOAUXQL4U4BKL6PKCBKXBUR2DXWNAW4A, seg_pos={0 0}, upload=497056]: Long tail cancellation due to optimal threshold Dec 29 06:25:08.472166458 (566h56m8.343249523s) [noise=true , address=145.79.227.44:4193 , piece_id=AFO4EOIFF6HTIHR7GDODVYAT3FDSZLCJKBFQY6VTOVXLERYTX33Q, seg_pos={0 0}, upload=497056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 06:25:08.472176657 (566h56m8.343259702s) [noise=true , address=145.79.227.44:4193 , piece_id=AFO4EOIFF6HTIHR7GDODVYAT3FDSZLCJKBFQY6VTOVXLERYTX33Q, seg_pos={0 0}, upload=497056]: Long tail cancellation due to optimal threshold Dec 29 06:25:08.472404331 (566h56m8.343487366s) [noise=true , address=45.91.116.13:4154 , piece_id=TEC2A2BYK6CLATG6ASPARLL6POUJZO3NRXCSA5BUCV7YGLPO5T4A, seg_pos={0 0}, upload=497056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:49840->45.91.116.13:4154: use of closed network connection; context canceled Dec 29 06:25:08.472414480 (566h56m8.343497525s) [noise=true , address=45.91.116.13:4154 , piece_id=TEC2A2BYK6CLATG6ASPARLL6POUJZO3NRXCSA5BUCV7YGLPO5T4A, seg_pos={0 0}, upload=497056]: Long tail cancellation due to optimal threshold Dec 29 06:25:08.472438226 (566h56m8.343521271s) [seg_pos={0 0}, upload=497056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:25:08.472577671 (566h56m8.343660716s) [upload=497056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:25:08.472598741 (566h56m8.343681796s) [upload=497056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:25:08.772903293 (566h56m8.643986338s) [upload=497056]: Done uploading segments... Dec 29 06:25:08.772924694 (566h56m8.644007739s) [upload=497056]: Upload finished. err: <nil> Duration: 30.002411061s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [XX.X.....X......XX......................X..XX...............] duration: 3.498740876s trace: https://jaeger.dev.storj.io/trace/20c51758eaefede7 FAIL (2025-12-29 06:25:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49881145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.X.....................X......................] duration: 36.892520128s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:25:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 36.892544032s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.XX.X.....X......XX......................X..XX..............] duration: 3.447218709s trace: https://jaeger.dev.storj.io/trace/20c51758eaefede7 FAIL (2025-12-29 06:24:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447248856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX....X.X.X..X...........X.X..X..X........X..X.....X.X.X.X..] duration: 7.914274313s trace: https://jaeger.dev.storj.io/trace/4fe0dafd18b5ecb8 FAIL (2025-12-29 06:24:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: SSMLUIZASNIIWZTOJI5CLYKXRPSDPZ6G4VJUNFTH4LCOI37HR7SA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: FK36HLFUOGU2XCDZMUXMTMKKBBWA2DBYF3E73YHSRY2STGUZR77A) context deadline exceeded ======LOGS====== Dec 29 06:22:46.393622320 (566h53m46.264705365s) [upload=497020]: Starting upload Dec 29 06:22:46.393700559 (566h53m46.264783594s) [upload=497020]: Done starting upload Dec 29 06:22:46.393765663 (566h53m46.264848708s) [upload=497020]: Uploading segments... Dec 29 06:22:46.394036389 (566h53m46.265119424s) [upload=497020]: Got next segment. Inline: false Dec 29 06:22:46.394087806 (566h53m46.265170841s) [upload=497020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:22:46.394092665 (566h53m46.26517571s) [upload=497020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:22:46.636378394 (566h53m46.507461429s) [seg_pos={0 0}, upload=497020]: Begin upload segment... Dec 29 06:22:46.636436544 (566h53m46.507519589s) [seg_pos={0 0}, upload=497020]: Done begin upload segment. Dec 29 06:22:46.636454228 (566h53m46.507537273s) [noise=true , address=145.79.227.15:4873 , piece_id=OOBKQJRWNLWRMMTCDYURCCOE54KBOUWFU4QU2OQT7CN67ZGJVFBA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.636477002 (566h53m46.507560037s) [noise=true , address=45.91.116.9:4262 , piece_id=SSMLUIZASNIIWZTOJI5CLYKXRPSDPZ6G4VJUNFTH4LCOI37HR7SA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.636557615 (566h53m46.50764065s) [noise=true , address=145.79.227.42:4123 , piece_id=HZHPCKO4QFJB43GTFWVUUJFL7VM5EALFH2FMRSDFICDQFGQTM62Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.637242449 (566h53m46.508325504s) [noise=true , address=77.90.31.21:4880 , piece_id=S4IUQL5Z2OIJNL6JOPSKZ5LCSCIP5OVULHTI5L6Q4HAN36CG5CGA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.638322225 (566h53m46.50940526s) [noise=true , address=77.90.31.18:4808 , piece_id=HSFN3HZ3TN6UZ5AWMPOTTY3CI7Q4SQIFQ67CHEOGEZXKZN3SPJGQ, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.639007400 (566h53m46.510090435s) [noise=true , address=145.79.227.36:4517 , piece_id=ZT244ZF2C2JI3YZ4CMLRDJNYTAVGLYPY3CLCL5CGZRL53QBLAXYA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.639514716 (566h53m46.510597751s) [noise=true , address=45.91.116.7:4012 , piece_id=QTXQTIDTFBNCEZ5AZTHJJPRWZCFVYVMIAXVPVJCILIY7QXN23RXA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.639989800 (566h53m46.511072845s) [noise=true , address=77.90.31.19:4844 , piece_id=65TMOXNITAILJNEJNMR7P2CLPB5ATSZGSEBCRJBWIA6XJ6IWZ6EA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.640566408 (566h53m46.511649453s) [noise=true , address=145.79.227.52:4446 , piece_id=GKAZGJMS3GQ4OYUJLWNI2UEZWONQRDCYFW72EDAGRZGAQKNTD6FQ, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.641894377 (566h53m46.512977422s) [noise=true , address=145.79.227.2:4114 , piece_id=NINQC52DWQT7C42IKM4NRQTBYJHRF6KHOZ4FX5S2ZRRXUI2VYUGA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.642429085 (566h53m46.51351212s) [noise=true , address=45.91.116.5:4300 , piece_id=24NHVXU7G7WZTHJWONFY7NDDFKXYNRMPLSNWQMVFK7NDWNGBHMAA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.643741464 (566h53m46.514824509s) [noise=true , address=145.79.227.7:4295 , piece_id=ONTJHINRDSAGDZG4ONLD3DTUIR5LFALGV5VMDQ5ZJDZH2DTARJ6Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.644198554 (566h53m46.515281599s) [noise=true , address=145.79.227.39:4014 , piece_id=S6LFZ3QUS46SBN47HQDBAE3A5MTN6RWWRMD2DW67LW7YLXG3R54Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.646167213 (566h53m46.517250248s) [noise=true , address=45.91.116.14:4190 , piece_id=FK36HLFUOGU2XCDZMUXMTMKKBBWA2DBYF3E73YHSRY2STGUZR77A, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.648422097 (566h53m46.519505132s) [noise=true , address=45.91.116.13:4153 , piece_id=OGBTEMR4ZYMTZSBLWPBICDPQHQAKVPGNAXXWUVCTVW52LF3GVS5Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.650695035 (566h53m46.52177808s) [noise=true , address=145.79.227.35:4482 , piece_id=TSAGGDB3TCDK3FTEQGCR6CCNU64FMVCCNY6QM52WFBF6PPY3VADA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:48.341930138 (566h53m48.213013183s) [upload=497020]: Next returned nil segment Dec 29 06:22:48.341946348 (566h53m48.213029384s) [upload=497020]: Waiting for error group managing segments... Dec 29 06:22:48.627850734 (566h53m48.498933769s) [noise=true , address=145.79.227.2:4114 , piece_id=NINQC52DWQT7C42IKM4NRQTBYJHRF6KHOZ4FX5S2ZRRXUI2VYUGA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.628722794 (566h53m48.499805839s) [noise=true , address=145.79.227.36:4517 , piece_id=ZT244ZF2C2JI3YZ4CMLRDJNYTAVGLYPY3CLCL5CGZRL53QBLAXYA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.639944070 (566h53m48.511027105s) [noise=true , address=145.79.227.39:4014 , piece_id=S6LFZ3QUS46SBN47HQDBAE3A5MTN6RWWRMD2DW67LW7YLXG3R54Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.646394323 (566h53m48.517477358s) [noise=true , address=145.79.227.52:4446 , piece_id=GKAZGJMS3GQ4OYUJLWNI2UEZWONQRDCYFW72EDAGRZGAQKNTD6FQ, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.656622749 (566h53m48.527705784s) [noise=true , address=145.79.227.15:4873 , piece_id=OOBKQJRWNLWRMMTCDYURCCOE54KBOUWFU4QU2OQT7CN67ZGJVFBA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.670709353 (566h53m48.541792388s) [noise=true , address=145.79.227.7:4295 , piece_id=ONTJHINRDSAGDZG4ONLD3DTUIR5LFALGV5VMDQ5ZJDZH2DTARJ6Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.680735944 (566h53m48.551818999s) [noise=true , address=145.79.227.42:4123 , piece_id=HZHPCKO4QFJB43GTFWVUUJFL7VM5EALFH2FMRSDFICDQFGQTM62Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.843054703 (566h53m48.714137748s) [noise=true , address=77.90.31.18:4808 , piece_id=HSFN3HZ3TN6UZ5AWMPOTTY3CI7Q4SQIFQ67CHEOGEZXKZN3SPJGQ, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.877818679 (566h53m48.748901714s) [noise=true , address=45.91.116.5:4300 , piece_id=24NHVXU7G7WZTHJWONFY7NDDFKXYNRMPLSNWQMVFK7NDWNGBHMAA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.890168676 (566h53m48.761251711s) [noise=true , address=45.91.116.9:4262 , piece_id=SSMLUIZASNIIWZTOJI5CLYKXRPSDPZ6G4VJUNFTH4LCOI37HR7SA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.914550792 (566h53m48.785633837s) [noise=true , address=45.91.116.14:4190 , piece_id=FK36HLFUOGU2XCDZMUXMTMKKBBWA2DBYF3E73YHSRY2STGUZR77A, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.944275824 (566h53m48.815358879s) [noise=true , address=77.90.31.21:4880 , piece_id=S4IUQL5Z2OIJNL6JOPSKZ5LCSCIP5OVULHTI5L6Q4HAN36CG5CGA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:49.540201696 (566h53m49.411284731s) [noise=true , address=77.90.31.19:4844 , piece_id=65TMOXNITAILJNEJNMR7P2CLPB5ATSZGSEBCRJBWIA6XJ6IWZ6EA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:49.540222024 (566h53m49.411305069s) [seg_pos={0 0}, upload=497020]: Segment reached optimal threshold of 13 pieces. Dec 29 06:22:49.540379524 (566h53m49.411462559s) [noise=true , address=145.79.227.35:4482 , piece_id=TSAGGDB3TCDK3FTEQGCR6CCNU64FMVCCNY6QM52WFBF6PPY3VADA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 06:22:49.540385345 (566h53m49.41146838s) [noise=true , address=145.79.227.35:4482 , piece_id=TSAGGDB3TCDK3FTEQGCR6CCNU64FMVCCNY6QM52WFBF6PPY3VADA, seg_pos={0 0}, upload=497020]: Long tail cancellation due to optimal threshold Dec 29 06:22:49.540447353 (566h53m49.411530398s) [noise=true , address=45.91.116.13:4153 , piece_id=OGBTEMR4ZYMTZSBLWPBICDPQHQAKVPGNAXXWUVCTVW52LF3GVS5Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 29 06:22:49.540455559 (566h53m49.411538594s) [noise=true , address=45.91.116.13:4153 , piece_id=OGBTEMR4ZYMTZSBLWPBICDPQHQAKVPGNAXXWUVCTVW52LF3GVS5Q, seg_pos={0 0}, upload=497020]: Long tail cancellation due to optimal threshold Dec 29 06:22:49.540496777 (566h53m49.411579812s) [noise=true , address=45.91.116.7:4012 , piece_id=QTXQTIDTFBNCEZ5AZTHJJPRWZCFVYVMIAXVPVJCILIY7QXN23RXA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 29 06:22:49.540504451 (566h53m49.411587486s) [noise=true , address=45.91.116.7:4012 , piece_id=QTXQTIDTFBNCEZ5AZTHJJPRWZCFVYVMIAXVPVJCILIY7QXN23RXA, seg_pos={0 0}, upload=497020]: Long tail cancellation due to optimal threshold Dec 29 06:22:49.540538055 (566h53m49.4116211s) [seg_pos={0 0}, upload=497020]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:22:49.540607287 (566h53m49.411690322s) [upload=497020]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:22:49.540612347 (566h53m49.411695382s) [upload=497020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:22:49.836562339 (566h53m49.707645374s) [upload=497020]: Done uploading segments... Dec 29 06:22:49.836579733 (566h53m49.707662768s) [upload=497020]: Upload finished. err: <nil> Duration: 7.914323177s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.X.....................X.....................] duration: 31.236480513s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:24:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 31.236498157s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX....X.X.X..X...........X.X..X..X........X..X.....X.X.X.X.] duration: 5.695975724s trace: https://jaeger.dev.storj.io/trace/4fe0dafd18b5ecb8 FAIL (2025-12-29 06:23:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: SSMLUIZASNIIWZTOJI5CLYKXRPSDPZ6G4VJUNFTH4LCOI37HR7SA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: FK36HLFUOGU2XCDZMUXMTMKKBBWA2DBYF3E73YHSRY2STGUZR77A) context deadline exceeded ======LOGS====== Dec 29 06:22:46.393622320 (566h53m46.264705365s) [upload=497020]: Starting upload Dec 29 06:22:46.393700559 (566h53m46.264783594s) [upload=497020]: Done starting upload Dec 29 06:22:46.393765663 (566h53m46.264848708s) [upload=497020]: Uploading segments... Dec 29 06:22:46.394036389 (566h53m46.265119424s) [upload=497020]: Got next segment. Inline: false Dec 29 06:22:46.394087806 (566h53m46.265170841s) [upload=497020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:22:46.394092665 (566h53m46.26517571s) [upload=497020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:22:46.636378394 (566h53m46.507461429s) [seg_pos={0 0}, upload=497020]: Begin upload segment... Dec 29 06:22:46.636436544 (566h53m46.507519589s) [seg_pos={0 0}, upload=497020]: Done begin upload segment. Dec 29 06:22:46.636454228 (566h53m46.507537273s) [noise=true , address=145.79.227.15:4873 , piece_id=OOBKQJRWNLWRMMTCDYURCCOE54KBOUWFU4QU2OQT7CN67ZGJVFBA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.636477002 (566h53m46.507560037s) [noise=true , address=45.91.116.9:4262 , piece_id=SSMLUIZASNIIWZTOJI5CLYKXRPSDPZ6G4VJUNFTH4LCOI37HR7SA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.636557615 (566h53m46.50764065s) [noise=true , address=145.79.227.42:4123 , piece_id=HZHPCKO4QFJB43GTFWVUUJFL7VM5EALFH2FMRSDFICDQFGQTM62Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.637242449 (566h53m46.508325504s) [noise=true , address=77.90.31.21:4880 , piece_id=S4IUQL5Z2OIJNL6JOPSKZ5LCSCIP5OVULHTI5L6Q4HAN36CG5CGA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.638322225 (566h53m46.50940526s) [noise=true , address=77.90.31.18:4808 , piece_id=HSFN3HZ3TN6UZ5AWMPOTTY3CI7Q4SQIFQ67CHEOGEZXKZN3SPJGQ, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.639007400 (566h53m46.510090435s) [noise=true , address=145.79.227.36:4517 , piece_id=ZT244ZF2C2JI3YZ4CMLRDJNYTAVGLYPY3CLCL5CGZRL53QBLAXYA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.639514716 (566h53m46.510597751s) [noise=true , address=45.91.116.7:4012 , piece_id=QTXQTIDTFBNCEZ5AZTHJJPRWZCFVYVMIAXVPVJCILIY7QXN23RXA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.639989800 (566h53m46.511072845s) [noise=true , address=77.90.31.19:4844 , piece_id=65TMOXNITAILJNEJNMR7P2CLPB5ATSZGSEBCRJBWIA6XJ6IWZ6EA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.640566408 (566h53m46.511649453s) [noise=true , address=145.79.227.52:4446 , piece_id=GKAZGJMS3GQ4OYUJLWNI2UEZWONQRDCYFW72EDAGRZGAQKNTD6FQ, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.641894377 (566h53m46.512977422s) [noise=true , address=145.79.227.2:4114 , piece_id=NINQC52DWQT7C42IKM4NRQTBYJHRF6KHOZ4FX5S2ZRRXUI2VYUGA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.642429085 (566h53m46.51351212s) [noise=true , address=45.91.116.5:4300 , piece_id=24NHVXU7G7WZTHJWONFY7NDDFKXYNRMPLSNWQMVFK7NDWNGBHMAA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.643741464 (566h53m46.514824509s) [noise=true , address=145.79.227.7:4295 , piece_id=ONTJHINRDSAGDZG4ONLD3DTUIR5LFALGV5VMDQ5ZJDZH2DTARJ6Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.644198554 (566h53m46.515281599s) [noise=true , address=145.79.227.39:4014 , piece_id=S6LFZ3QUS46SBN47HQDBAE3A5MTN6RWWRMD2DW67LW7YLXG3R54Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.646167213 (566h53m46.517250248s) [noise=true , address=45.91.116.14:4190 , piece_id=FK36HLFUOGU2XCDZMUXMTMKKBBWA2DBYF3E73YHSRY2STGUZR77A, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.648422097 (566h53m46.519505132s) [noise=true , address=45.91.116.13:4153 , piece_id=OGBTEMR4ZYMTZSBLWPBICDPQHQAKVPGNAXXWUVCTVW52LF3GVS5Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.650695035 (566h53m46.52177808s) [noise=true , address=145.79.227.35:4482 , piece_id=TSAGGDB3TCDK3FTEQGCR6CCNU64FMVCCNY6QM52WFBF6PPY3VADA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:48.341930138 (566h53m48.213013183s) [upload=497020]: Next returned nil segment Dec 29 06:22:48.341946348 (566h53m48.213029384s) [upload=497020]: Waiting for error group managing segments... Dec 29 06:22:48.627850734 (566h53m48.498933769s) [noise=true , address=145.79.227.2:4114 , piece_id=NINQC52DWQT7C42IKM4NRQTBYJHRF6KHOZ4FX5S2ZRRXUI2VYUGA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.628722794 (566h53m48.499805839s) [noise=true , address=145.79.227.36:4517 , piece_id=ZT244ZF2C2JI3YZ4CMLRDJNYTAVGLYPY3CLCL5CGZRL53QBLAXYA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.639944070 (566h53m48.511027105s) [noise=true , address=145.79.227.39:4014 , piece_id=S6LFZ3QUS46SBN47HQDBAE3A5MTN6RWWRMD2DW67LW7YLXG3R54Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.646394323 (566h53m48.517477358s) [noise=true , address=145.79.227.52:4446 , piece_id=GKAZGJMS3GQ4OYUJLWNI2UEZWONQRDCYFW72EDAGRZGAQKNTD6FQ, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.656622749 (566h53m48.527705784s) [noise=true , address=145.79.227.15:4873 , piece_id=OOBKQJRWNLWRMMTCDYURCCOE54KBOUWFU4QU2OQT7CN67ZGJVFBA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.670709353 (566h53m48.541792388s) [noise=true , address=145.79.227.7:4295 , piece_id=ONTJHINRDSAGDZG4ONLD3DTUIR5LFALGV5VMDQ5ZJDZH2DTARJ6Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.680735944 (566h53m48.551818999s) [noise=true , address=145.79.227.42:4123 , piece_id=HZHPCKO4QFJB43GTFWVUUJFL7VM5EALFH2FMRSDFICDQFGQTM62Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.843054703 (566h53m48.714137748s) [noise=true , address=77.90.31.18:4808 , piece_id=HSFN3HZ3TN6UZ5AWMPOTTY3CI7Q4SQIFQ67CHEOGEZXKZN3SPJGQ, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.877818679 (566h53m48.748901714s) [noise=true , address=45.91.116.5:4300 , piece_id=24NHVXU7G7WZTHJWONFY7NDDFKXYNRMPLSNWQMVFK7NDWNGBHMAA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.890168676 (566h53m48.761251711s) [noise=true , address=45.91.116.9:4262 , piece_id=SSMLUIZASNIIWZTOJI5CLYKXRPSDPZ6G4VJUNFTH4LCOI37HR7SA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.914550792 (566h53m48.785633837s) [noise=true , address=45.91.116.14:4190 , piece_id=FK36HLFUOGU2XCDZMUXMTMKKBBWA2DBYF3E73YHSRY2STGUZR77A, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.944275824 (566h53m48.815358879s) [noise=true , address=77.90.31.21:4880 , piece_id=S4IUQL5Z2OIJNL6JOPSKZ5LCSCIP5OVULHTI5L6Q4HAN36CG5CGA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:49.540201696 (566h53m49.411284731s) [noise=true , address=77.90.31.19:4844 , piece_id=65TMOXNITAILJNEJNMR7P2CLPB5ATSZGSEBCRJBWIA6XJ6IWZ6EA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:49.540222024 (566h53m49.411305069s) [seg_pos={0 0}, upload=497020]: Segment reached optimal threshold of 13 pieces. Dec 29 06:22:49.540379524 (566h53m49.411462559s) [noise=true , address=145.79.227.35:4482 , piece_id=TSAGGDB3TCDK3FTEQGCR6CCNU64FMVCCNY6QM52WFBF6PPY3VADA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 06:22:49.540385345 (566h53m49.41146838s) [noise=true , address=145.79.227.35:4482 , piece_id=TSAGGDB3TCDK3FTEQGCR6CCNU64FMVCCNY6QM52WFBF6PPY3VADA, seg_pos={0 0}, upload=497020]: Long tail cancellation due to optimal threshold Dec 29 06:22:49.540447353 (566h53m49.411530398s) [noise=true , address=45.91.116.13:4153 , piece_id=OGBTEMR4ZYMTZSBLWPBICDPQHQAKVPGNAXXWUVCTVW52LF3GVS5Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 29 06:22:49.540455559 (566h53m49.411538594s) [noise=true , address=45.91.116.13:4153 , piece_id=OGBTEMR4ZYMTZSBLWPBICDPQHQAKVPGNAXXWUVCTVW52LF3GVS5Q, seg_pos={0 0}, upload=497020]: Long tail cancellation due to optimal threshold Dec 29 06:22:49.540496777 (566h53m49.411579812s) [noise=true , address=45.91.116.7:4012 , piece_id=QTXQTIDTFBNCEZ5AZTHJJPRWZCFVYVMIAXVPVJCILIY7QXN23RXA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 29 06:22:49.540504451 (566h53m49.411587486s) [noise=true , address=45.91.116.7:4012 , piece_id=QTXQTIDTFBNCEZ5AZTHJJPRWZCFVYVMIAXVPVJCILIY7QXN23RXA, seg_pos={0 0}, upload=497020]: Long tail cancellation due to optimal threshold Dec 29 06:22:49.540538055 (566h53m49.4116211s) [seg_pos={0 0}, upload=497020]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:22:49.540607287 (566h53m49.411690322s) [upload=497020]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:22:49.540612347 (566h53m49.411695382s) [upload=497020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:22:49.836562339 (566h53m49.707645374s) [upload=497020]: Done uploading segments... Dec 29 06:22:49.836579733 (566h53m49.707662768s) [upload=497020]: Upload finished. err: <nil> Duration: 5.696006461s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..XX.X.....X......XX......................X..XX.............] duration: 3.619590649s trace: https://jaeger.dev.storj.io/trace/20c51758eaefede7 FAIL (2025-12-29 06:23:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.61961785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.X.....................X....................] duration: 43.145966952s trace: https://jaeger.dev.storj.io/trace/531b9de0e13ca424 FAIL (2025-12-29 06:23:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.242961431 (566h37m14.114044476s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243369618 (566h37m14.114452664s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.243813955 (566h37m14.11489699s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244240717 (566h37m14.115323752s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.244685895 (566h37m14.11576894s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245126615 (566h37m14.11620966s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.245604243 (566h37m14.116687288s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246063979 (566h37m14.117147024s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.246710029 (566h37m14.117793064s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247135639 (566h37m14.118218675s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247505474 (566h37m14.118588509s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.247876641 (566h37m14.118959676s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248169879 (566h37m14.119252924s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.248808865 (566h37m14.119891911s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:14.249003226 (566h37m14.120086261s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Uploading piece... Dec 29 06:06:15.700101304 (566h37m15.571184339s) [upload=496783]: Next returned nil segment Dec 29 06:06:15.700115571 (566h37m15.571198606s) [upload=496783]: Waiting for error group managing segments... Dec 29 06:06:16.099772291 (566h37m15.970855336s) [noise=true , address=145.79.227.17:4999 , piece_id=VO7FAEMEH4CCN6H7XI4DDPJPRSRDOYP3S4S7QLPSUICEVFDAAXQA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.101331149 (566h37m15.972414194s) [noise=true , address=145.79.227.4:4193 , piece_id=RGQGLBTOTWTP37XG7FUZ5TXNVDMHC7UW4GQD7EYQSPAWDLOP5AEA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.300118551 (566h37m16.171201596s) [noise=true , address=77.90.31.17:4792 , piece_id=QAK7RHFUTSDEKCKMT4KVPQ5SIANQSAMVVHPBMRCS7N5M35QE4I4A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.301918819 (566h37m16.173001864s) [noise=true , address=77.90.31.18:4832 , piece_id=JLBPBJ47HLSC3VMGFWXH6X4CFUHPLEDICRRLB2MAXRY4XAQIWLCQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.338382662 (566h37m16.209465697s) [noise=true , address=77.90.31.19:4853 , piece_id=PUQ74M2ZFJOIHZLPU2KPUFFN3OLQMWMA3Z2JPHCOUNLJQCA5RN2Q, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.389432007 (566h37m16.260515052s) [noise=true , address=77.90.31.16:4744 , piece_id=ADSU475GQWPNZFHVY3V4NLM2YSGGQKQZGW75M6YWW4YHP3TIS4CQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.394666085 (566h37m16.26574913s) [noise=true , address=77.90.31.21:4904 , piece_id=GHFP4PC4MIOR5QMA3GS3X256HEHDN7CU5SVSBQ6O2QSZYY5DUIIQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.426250236 (566h37m16.297333281s) [noise=true , address=145.79.227.12:4776 , piece_id=XBEZA4XVDTHQSRVA4HPWPZFX3DMZET2WZMICSGOFVPDESTC36YJQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.474507946 (566h37m16.345590991s) [noise=true , address=45.91.116.8:4052 , piece_id=ZORU6U3RUQRRF4FGFV4FHCOMGYWA624ZTKOUOEFKLA7YUXKGJFYA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:16.480417338 (566h37m16.351500384s) [noise=true , address=77.90.31.14:4671 , piece_id=JWOC4P2RVKYC6GK33HOTRNL2CM73DSJ6XNEW2SZ4TTKNV2X2FE3A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.011859447 (566h37m16.882942492s) [noise=true , address=145.79.227.30:4788 , piece_id=FKXWBXKDLO2UQKS3EWJBIV7V6KOSZ3EMQJDT6FLQERJIC2LR3CHQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030017915 (566h37m16.90110096s) [noise=true , address=145.79.227.43:4191 , piece_id=G6HHWMIYUAV6TK4OZLIAPN7QMC4D4GIZOEATGQKHSJVYGTXERNBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.030528596 (566h37m16.901611631s) [noise=true , address=145.79.227.40:4058 , piece_id=WQ5NLXNT4SQZ55DXTFWB73XTTOYGEEAOIQPKH6CCYDQGMJ3M4BVA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050202970 (566h37m16.921286005s) [noise=true , address=145.79.227.44:4225 , piece_id=JJOIFEI3XEPCOV4LNXHDILOZRZP4WHZEHUUNY3OPZRBA3AYTY22A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: <nil> Dec 29 06:06:17.050224872 (566h37m16.921307917s) [seg_pos={0 0}, upload=496783]: Segment reached optimal threshold of 14 pieces. Dec 29 06:06:17.050565810 (566h37m16.921648875s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50658->64.49.13.169:4715: use of closed network connection Dec 29 06:06:17.050654940 (566h37m16.921737985s) [noise=true , address=64.49.13.169:4715 , piece_id=75RKGTJ7T3PPIYSNJFXPCXREDI5A5ANOBRGXVOVMGC35376E7N6A, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.050920837 (566h37m16.922003882s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.050941786 (566h37m16.922024821s) [noise=true , address=193.149.164.143:4694 , piece_id=H55F623SNDXUTQHXYDLFMAQIM4RULKK7BCMBCII7KSXEPRBGKJPA, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051125416 (566h37m16.922208461s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051133071 (566h37m16.922216106s) [noise=true , address=145.79.227.49:4358 , piece_id=TSVFDTRQOK4ADVORC4ODYYVF67SD3RQ4KHV2BP75UO44YAIM75DQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051298555 (566h37m16.92238159s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 06:06:17.051306311 (566h37m16.922389356s) [noise=true , address=145.79.227.33:4898 , piece_id=DBNL4CPZ4K3C4HRYOVEFL2ABLTLXQ67JCYUE7NP3LIPZDHQAETBQ, seg_pos={0 0}, upload=496783]: Long tail cancellation due to optimal threshold Dec 29 06:06:17.051326308 (566h37m16.922409353s) [seg_pos={0 0}, upload=496783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 06:06:17.051397364 (566h37m16.922480439s) [upload=496783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:06:17.051407403 (566h37m16.922490479s) [upload=496783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:06:17.364457360 (566h37m17.235540415s) [upload=496783]: Done uploading segments... Dec 29 06:06:17.364477588 (566h37m17.235560633s) [upload=496783]: Upload finished. err: <nil> Duration: 43.146010735s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...XX.X.....X......XX......................X..XX............] duration: 3.501713222s trace: https://jaeger.dev.storj.io/trace/20c51758eaefede7 FAIL (2025-12-29 06:23:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501776972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXX....X.X.X..X...........X.X..X..X........X..X.....X.X.X.X] duration: 30.002937525s trace: https://jaeger.dev.storj.io/trace/4fe0dafd18b5ecb8 FAIL (2025-12-29 06:23:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: SSMLUIZASNIIWZTOJI5CLYKXRPSDPZ6G4VJUNFTH4LCOI37HR7SA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: FK36HLFUOGU2XCDZMUXMTMKKBBWA2DBYF3E73YHSRY2STGUZR77A) context deadline exceeded ======LOGS====== Dec 29 06:22:46.393622320 (566h53m46.264705365s) [upload=497020]: Starting upload Dec 29 06:22:46.393700559 (566h53m46.264783594s) [upload=497020]: Done starting upload Dec 29 06:22:46.393765663 (566h53m46.264848708s) [upload=497020]: Uploading segments... Dec 29 06:22:46.394036389 (566h53m46.265119424s) [upload=497020]: Got next segment. Inline: false Dec 29 06:22:46.394087806 (566h53m46.265170841s) [upload=497020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:22:46.394092665 (566h53m46.26517571s) [upload=497020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:22:46.636378394 (566h53m46.507461429s) [seg_pos={0 0}, upload=497020]: Begin upload segment... Dec 29 06:22:46.636436544 (566h53m46.507519589s) [seg_pos={0 0}, upload=497020]: Done begin upload segment. Dec 29 06:22:46.636454228 (566h53m46.507537273s) [noise=true , address=145.79.227.15:4873 , piece_id=OOBKQJRWNLWRMMTCDYURCCOE54KBOUWFU4QU2OQT7CN67ZGJVFBA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.636477002 (566h53m46.507560037s) [noise=true , address=45.91.116.9:4262 , piece_id=SSMLUIZASNIIWZTOJI5CLYKXRPSDPZ6G4VJUNFTH4LCOI37HR7SA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.636557615 (566h53m46.50764065s) [noise=true , address=145.79.227.42:4123 , piece_id=HZHPCKO4QFJB43GTFWVUUJFL7VM5EALFH2FMRSDFICDQFGQTM62Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.637242449 (566h53m46.508325504s) [noise=true , address=77.90.31.21:4880 , piece_id=S4IUQL5Z2OIJNL6JOPSKZ5LCSCIP5OVULHTI5L6Q4HAN36CG5CGA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.638322225 (566h53m46.50940526s) [noise=true , address=77.90.31.18:4808 , piece_id=HSFN3HZ3TN6UZ5AWMPOTTY3CI7Q4SQIFQ67CHEOGEZXKZN3SPJGQ, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.639007400 (566h53m46.510090435s) [noise=true , address=145.79.227.36:4517 , piece_id=ZT244ZF2C2JI3YZ4CMLRDJNYTAVGLYPY3CLCL5CGZRL53QBLAXYA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.639514716 (566h53m46.510597751s) [noise=true , address=45.91.116.7:4012 , piece_id=QTXQTIDTFBNCEZ5AZTHJJPRWZCFVYVMIAXVPVJCILIY7QXN23RXA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.639989800 (566h53m46.511072845s) [noise=true , address=77.90.31.19:4844 , piece_id=65TMOXNITAILJNEJNMR7P2CLPB5ATSZGSEBCRJBWIA6XJ6IWZ6EA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.640566408 (566h53m46.511649453s) [noise=true , address=145.79.227.52:4446 , piece_id=GKAZGJMS3GQ4OYUJLWNI2UEZWONQRDCYFW72EDAGRZGAQKNTD6FQ, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.641894377 (566h53m46.512977422s) [noise=true , address=145.79.227.2:4114 , piece_id=NINQC52DWQT7C42IKM4NRQTBYJHRF6KHOZ4FX5S2ZRRXUI2VYUGA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.642429085 (566h53m46.51351212s) [noise=true , address=45.91.116.5:4300 , piece_id=24NHVXU7G7WZTHJWONFY7NDDFKXYNRMPLSNWQMVFK7NDWNGBHMAA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.643741464 (566h53m46.514824509s) [noise=true , address=145.79.227.7:4295 , piece_id=ONTJHINRDSAGDZG4ONLD3DTUIR5LFALGV5VMDQ5ZJDZH2DTARJ6Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.644198554 (566h53m46.515281599s) [noise=true , address=145.79.227.39:4014 , piece_id=S6LFZ3QUS46SBN47HQDBAE3A5MTN6RWWRMD2DW67LW7YLXG3R54Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.646167213 (566h53m46.517250248s) [noise=true , address=45.91.116.14:4190 , piece_id=FK36HLFUOGU2XCDZMUXMTMKKBBWA2DBYF3E73YHSRY2STGUZR77A, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.648422097 (566h53m46.519505132s) [noise=true , address=45.91.116.13:4153 , piece_id=OGBTEMR4ZYMTZSBLWPBICDPQHQAKVPGNAXXWUVCTVW52LF3GVS5Q, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:46.650695035 (566h53m46.52177808s) [noise=true , address=145.79.227.35:4482 , piece_id=TSAGGDB3TCDK3FTEQGCR6CCNU64FMVCCNY6QM52WFBF6PPY3VADA, seg_pos={0 0}, upload=497020]: Uploading piece... Dec 29 06:22:48.341930138 (566h53m48.213013183s) [upload=497020]: Next returned nil segment Dec 29 06:22:48.341946348 (566h53m48.213029384s) [upload=497020]: Waiting for error group managing segments... Dec 29 06:22:48.627850734 (566h53m48.498933769s) [noise=true , address=145.79.227.2:4114 , piece_id=NINQC52DWQT7C42IKM4NRQTBYJHRF6KHOZ4FX5S2ZRRXUI2VYUGA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.628722794 (566h53m48.499805839s) [noise=true , address=145.79.227.36:4517 , piece_id=ZT244ZF2C2JI3YZ4CMLRDJNYTAVGLYPY3CLCL5CGZRL53QBLAXYA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.639944070 (566h53m48.511027105s) [noise=true , address=145.79.227.39:4014 , piece_id=S6LFZ3QUS46SBN47HQDBAE3A5MTN6RWWRMD2DW67LW7YLXG3R54Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.646394323 (566h53m48.517477358s) [noise=true , address=145.79.227.52:4446 , piece_id=GKAZGJMS3GQ4OYUJLWNI2UEZWONQRDCYFW72EDAGRZGAQKNTD6FQ, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.656622749 (566h53m48.527705784s) [noise=true , address=145.79.227.15:4873 , piece_id=OOBKQJRWNLWRMMTCDYURCCOE54KBOUWFU4QU2OQT7CN67ZGJVFBA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.670709353 (566h53m48.541792388s) [noise=true , address=145.79.227.7:4295 , piece_id=ONTJHINRDSAGDZG4ONLD3DTUIR5LFALGV5VMDQ5ZJDZH2DTARJ6Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.680735944 (566h53m48.551818999s) [noise=true , address=145.79.227.42:4123 , piece_id=HZHPCKO4QFJB43GTFWVUUJFL7VM5EALFH2FMRSDFICDQFGQTM62Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.843054703 (566h53m48.714137748s) [noise=true , address=77.90.31.18:4808 , piece_id=HSFN3HZ3TN6UZ5AWMPOTTY3CI7Q4SQIFQ67CHEOGEZXKZN3SPJGQ, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.877818679 (566h53m48.748901714s) [noise=true , address=45.91.116.5:4300 , piece_id=24NHVXU7G7WZTHJWONFY7NDDFKXYNRMPLSNWQMVFK7NDWNGBHMAA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.890168676 (566h53m48.761251711s) [noise=true , address=45.91.116.9:4262 , piece_id=SSMLUIZASNIIWZTOJI5CLYKXRPSDPZ6G4VJUNFTH4LCOI37HR7SA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.914550792 (566h53m48.785633837s) [noise=true , address=45.91.116.14:4190 , piece_id=FK36HLFUOGU2XCDZMUXMTMKKBBWA2DBYF3E73YHSRY2STGUZR77A, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:48.944275824 (566h53m48.815358879s) [noise=true , address=77.90.31.21:4880 , piece_id=S4IUQL5Z2OIJNL6JOPSKZ5LCSCIP5OVULHTI5L6Q4HAN36CG5CGA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:49.540201696 (566h53m49.411284731s) [noise=true , address=77.90.31.19:4844 , piece_id=65TMOXNITAILJNEJNMR7P2CLPB5ATSZGSEBCRJBWIA6XJ6IWZ6EA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: <nil> Dec 29 06:22:49.540222024 (566h53m49.411305069s) [seg_pos={0 0}, upload=497020]: Segment reached optimal threshold of 13 pieces. Dec 29 06:22:49.540379524 (566h53m49.411462559s) [noise=true , address=145.79.227.35:4482 , piece_id=TSAGGDB3TCDK3FTEQGCR6CCNU64FMVCCNY6QM52WFBF6PPY3VADA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 06:22:49.540385345 (566h53m49.41146838s) [noise=true , address=145.79.227.35:4482 , piece_id=TSAGGDB3TCDK3FTEQGCR6CCNU64FMVCCNY6QM52WFBF6PPY3VADA, seg_pos={0 0}, upload=497020]: Long tail cancellation due to optimal threshold Dec 29 06:22:49.540447353 (566h53m49.411530398s) [noise=true , address=45.91.116.13:4153 , piece_id=OGBTEMR4ZYMTZSBLWPBICDPQHQAKVPGNAXXWUVCTVW52LF3GVS5Q, seg_pos={0 0}, upload=497020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 29 06:22:49.540455559 (566h53m49.411538594s) [noise=true , address=45.91.116.13:4153 , piece_id=OGBTEMR4ZYMTZSBLWPBICDPQHQAKVPGNAXXWUVCTVW52LF3GVS5Q, seg_pos={0 0}, upload=497020]: Long tail cancellation due to optimal threshold Dec 29 06:22:49.540496777 (566h53m49.411579812s) [noise=true , address=45.91.116.7:4012 , piece_id=QTXQTIDTFBNCEZ5AZTHJJPRWZCFVYVMIAXVPVJCILIY7QXN23RXA, seg_pos={0 0}, upload=497020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 29 06:22:49.540504451 (566h53m49.411587486s) [noise=true , address=45.91.116.7:4012 , piece_id=QTXQTIDTFBNCEZ5AZTHJJPRWZCFVYVMIAXVPVJCILIY7QXN23RXA, seg_pos={0 0}, upload=497020]: Long tail cancellation due to optimal threshold Dec 29 06:22:49.540538055 (566h53m49.4116211s) [seg_pos={0 0}, upload=497020]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:22:49.540607287 (566h53m49.411690322s) [upload=497020]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:22:49.540612347 (566h53m49.411695382s) [upload=497020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:22:49.836562339 (566h53m49.707645374s) [upload=497020]: Done uploading segments... Dec 29 06:22:49.836579733 (566h53m49.707662768s) [upload=497020]: Upload finished. err: <nil> Duration: 30.002951252s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)