Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X...] duration: 5.821700358s trace: https://jaeger.dev.storj.io/trace/645f8a46074b6d72 FAIL (2025-12-27 16:56:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA) context deadline exceeded ======LOGS====== Dec 27 16:54:09.395244253 (529h25m9.266327288s) [upload=464328]: Starting upload Dec 27 16:54:09.395315138 (529h25m9.266398183s) [upload=464328]: Done starting upload Dec 27 16:54:09.395538925 (529h25m9.26662197s) [upload=464328]: Uploading segments... Dec 27 16:54:09.395664463 (529h25m9.266747518s) [upload=464328]: Got next segment. Inline: false Dec 27 16:54:09.395729166 (529h25m9.266812201s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:54:09.395736420 (529h25m9.266819465s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:54:09.640946212 (529h25m9.512029257s) [seg_pos={0 0}, upload=464328]: Begin upload segment... Dec 27 16:54:09.641002107 (529h25m9.512085153s) [seg_pos={0 0}, upload=464328]: Done begin upload segment. Dec 27 16:54:09.641033758 (529h25m9.512116823s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641104863 (529h25m9.512187898s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641605948 (529h25m9.512688983s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641793555 (529h25m9.51287659s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642172406 (529h25m9.513255441s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642571036 (529h25m9.513654071s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643041541 (529h25m9.514124576s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643124990 (529h25m9.514208035s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643844540 (529h25m9.514927575s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643909765 (529h25m9.51499281s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644564812 (529h25m9.515647847s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644616149 (529h25m9.515699184s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645316272 (529h25m9.516399307s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645380525 (529h25m9.51646357s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.646264679 (529h25m9.517347714s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.647019506 (529h25m9.518102541s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:11.251571098 (529h25m11.122654133s) [upload=464328]: Next returned nil segment Dec 27 16:54:11.251584153 (529h25m11.122667198s) [upload=464328]: Waiting for error group managing segments... Dec 27 16:54:11.635070998 (529h25m11.506154043s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.637746432 (529h25m11.508829467s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.642795406 (529h25m11.513878462s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.646797187 (529h25m11.517880222s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.648722423 (529h25m11.519805468s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.653115148 (529h25m11.524198183s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.669518742 (529h25m11.540601777s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.673846613 (529h25m11.544929658s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.682323394 (529h25m11.553406429s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.939294221 (529h25m11.810377266s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.959324092 (529h25m11.830407127s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.042123345 (529h25m11.91320639s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409785817 (529h25m12.280868883s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409842215 (529h25m12.28092526s) [seg_pos={0 0}, upload=464328]: Segment reached optimal threshold of 13 pieces. Dec 27 16:54:12.410069919 (529h25m12.281152974s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410079758 (529h25m12.281162803s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410393385 (529h25m12.28147643s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410409776 (529h25m12.281492821s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410432419 (529h25m12.281515464s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410449933 (529h25m12.281532968s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410468418 (529h25m12.281551463s) [seg_pos={0 0}, upload=464328]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:54:12.410589569 (529h25m12.281672654s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:54:12.410597073 (529h25m12.281680108s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:54:12.695997364 (529h25m12.567080409s) [upload=464328]: Done uploading segments... Dec 27 16:54:12.696056086 (529h25m12.567139121s) [upload=464328]: Upload finished. err: <nil> Duration: 5.821716018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X..] duration: 5.719749164s trace: https://jaeger.dev.storj.io/trace/645f8a46074b6d72 FAIL (2025-12-27 16:55:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA) context deadline exceeded ======LOGS====== Dec 27 16:54:09.395244253 (529h25m9.266327288s) [upload=464328]: Starting upload Dec 27 16:54:09.395315138 (529h25m9.266398183s) [upload=464328]: Done starting upload Dec 27 16:54:09.395538925 (529h25m9.26662197s) [upload=464328]: Uploading segments... Dec 27 16:54:09.395664463 (529h25m9.266747518s) [upload=464328]: Got next segment. Inline: false Dec 27 16:54:09.395729166 (529h25m9.266812201s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:54:09.395736420 (529h25m9.266819465s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:54:09.640946212 (529h25m9.512029257s) [seg_pos={0 0}, upload=464328]: Begin upload segment... Dec 27 16:54:09.641002107 (529h25m9.512085153s) [seg_pos={0 0}, upload=464328]: Done begin upload segment. Dec 27 16:54:09.641033758 (529h25m9.512116823s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641104863 (529h25m9.512187898s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641605948 (529h25m9.512688983s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641793555 (529h25m9.51287659s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642172406 (529h25m9.513255441s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642571036 (529h25m9.513654071s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643041541 (529h25m9.514124576s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643124990 (529h25m9.514208035s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643844540 (529h25m9.514927575s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643909765 (529h25m9.51499281s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644564812 (529h25m9.515647847s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644616149 (529h25m9.515699184s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645316272 (529h25m9.516399307s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645380525 (529h25m9.51646357s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.646264679 (529h25m9.517347714s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.647019506 (529h25m9.518102541s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:11.251571098 (529h25m11.122654133s) [upload=464328]: Next returned nil segment Dec 27 16:54:11.251584153 (529h25m11.122667198s) [upload=464328]: Waiting for error group managing segments... Dec 27 16:54:11.635070998 (529h25m11.506154043s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.637746432 (529h25m11.508829467s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.642795406 (529h25m11.513878462s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.646797187 (529h25m11.517880222s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.648722423 (529h25m11.519805468s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.653115148 (529h25m11.524198183s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.669518742 (529h25m11.540601777s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.673846613 (529h25m11.544929658s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.682323394 (529h25m11.553406429s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.939294221 (529h25m11.810377266s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.959324092 (529h25m11.830407127s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.042123345 (529h25m11.91320639s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409785817 (529h25m12.280868883s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409842215 (529h25m12.28092526s) [seg_pos={0 0}, upload=464328]: Segment reached optimal threshold of 13 pieces. Dec 27 16:54:12.410069919 (529h25m12.281152974s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410079758 (529h25m12.281162803s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410393385 (529h25m12.28147643s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410409776 (529h25m12.281492821s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410432419 (529h25m12.281515464s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410449933 (529h25m12.281532968s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410468418 (529h25m12.281551463s) [seg_pos={0 0}, upload=464328]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:54:12.410589569 (529h25m12.281672654s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:54:12.410597073 (529h25m12.281680108s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:54:12.695997364 (529h25m12.567080409s) [upload=464328]: Done uploading segments... Dec 27 16:54:12.696056086 (529h25m12.567139121s) [upload=464328]: Upload finished. err: <nil> Duration: 5.719789791s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X.....X...X.............................X..............X..] duration: 4.329102656s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 16:55:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.329127694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X.....X...X.............................X..............X.] duration: 5.001339081s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 16:55:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001368597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.] duration: 5.668958241s trace: https://jaeger.dev.storj.io/trace/645f8a46074b6d72 FAIL (2025-12-27 16:54:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA) context deadline exceeded ======LOGS====== Dec 27 16:54:09.395244253 (529h25m9.266327288s) [upload=464328]: Starting upload Dec 27 16:54:09.395315138 (529h25m9.266398183s) [upload=464328]: Done starting upload Dec 27 16:54:09.395538925 (529h25m9.26662197s) [upload=464328]: Uploading segments... Dec 27 16:54:09.395664463 (529h25m9.266747518s) [upload=464328]: Got next segment. Inline: false Dec 27 16:54:09.395729166 (529h25m9.266812201s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:54:09.395736420 (529h25m9.266819465s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:54:09.640946212 (529h25m9.512029257s) [seg_pos={0 0}, upload=464328]: Begin upload segment... Dec 27 16:54:09.641002107 (529h25m9.512085153s) [seg_pos={0 0}, upload=464328]: Done begin upload segment. Dec 27 16:54:09.641033758 (529h25m9.512116823s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641104863 (529h25m9.512187898s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641605948 (529h25m9.512688983s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641793555 (529h25m9.51287659s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642172406 (529h25m9.513255441s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642571036 (529h25m9.513654071s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643041541 (529h25m9.514124576s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643124990 (529h25m9.514208035s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643844540 (529h25m9.514927575s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643909765 (529h25m9.51499281s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644564812 (529h25m9.515647847s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644616149 (529h25m9.515699184s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645316272 (529h25m9.516399307s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645380525 (529h25m9.51646357s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.646264679 (529h25m9.517347714s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.647019506 (529h25m9.518102541s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:11.251571098 (529h25m11.122654133s) [upload=464328]: Next returned nil segment Dec 27 16:54:11.251584153 (529h25m11.122667198s) [upload=464328]: Waiting for error group managing segments... Dec 27 16:54:11.635070998 (529h25m11.506154043s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.637746432 (529h25m11.508829467s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.642795406 (529h25m11.513878462s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.646797187 (529h25m11.517880222s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.648722423 (529h25m11.519805468s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.653115148 (529h25m11.524198183s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.669518742 (529h25m11.540601777s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.673846613 (529h25m11.544929658s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.682323394 (529h25m11.553406429s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.939294221 (529h25m11.810377266s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.959324092 (529h25m11.830407127s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.042123345 (529h25m11.91320639s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409785817 (529h25m12.280868883s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409842215 (529h25m12.28092526s) [seg_pos={0 0}, upload=464328]: Segment reached optimal threshold of 13 pieces. Dec 27 16:54:12.410069919 (529h25m12.281152974s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410079758 (529h25m12.281162803s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410393385 (529h25m12.28147643s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410409776 (529h25m12.281492821s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410432419 (529h25m12.281515464s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410449933 (529h25m12.281532968s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410468418 (529h25m12.281551463s) [seg_pos={0 0}, upload=464328]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:54:12.410589569 (529h25m12.281672654s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:54:12.410597073 (529h25m12.281680108s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:54:12.695997364 (529h25m12.567080409s) [upload=464328]: Done uploading segments... Dec 27 16:54:12.696056086 (529h25m12.567139121s) [upload=464328]: Upload finished. err: <nil> Duration: 5.668982407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X] duration: 30.002780753s trace: https://jaeger.dev.storj.io/trace/645f8a46074b6d72 FAIL (2025-12-27 16:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA) context deadline exceeded ======LOGS====== Dec 27 16:54:09.395244253 (529h25m9.266327288s) [upload=464328]: Starting upload Dec 27 16:54:09.395315138 (529h25m9.266398183s) [upload=464328]: Done starting upload Dec 27 16:54:09.395538925 (529h25m9.26662197s) [upload=464328]: Uploading segments... Dec 27 16:54:09.395664463 (529h25m9.266747518s) [upload=464328]: Got next segment. Inline: false Dec 27 16:54:09.395729166 (529h25m9.266812201s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:54:09.395736420 (529h25m9.266819465s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:54:09.640946212 (529h25m9.512029257s) [seg_pos={0 0}, upload=464328]: Begin upload segment... Dec 27 16:54:09.641002107 (529h25m9.512085153s) [seg_pos={0 0}, upload=464328]: Done begin upload segment. Dec 27 16:54:09.641033758 (529h25m9.512116823s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641104863 (529h25m9.512187898s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641605948 (529h25m9.512688983s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641793555 (529h25m9.51287659s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642172406 (529h25m9.513255441s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642571036 (529h25m9.513654071s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643041541 (529h25m9.514124576s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643124990 (529h25m9.514208035s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643844540 (529h25m9.514927575s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643909765 (529h25m9.51499281s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644564812 (529h25m9.515647847s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644616149 (529h25m9.515699184s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645316272 (529h25m9.516399307s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645380525 (529h25m9.51646357s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.646264679 (529h25m9.517347714s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.647019506 (529h25m9.518102541s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:11.251571098 (529h25m11.122654133s) [upload=464328]: Next returned nil segment Dec 27 16:54:11.251584153 (529h25m11.122667198s) [upload=464328]: Waiting for error group managing segments... Dec 27 16:54:11.635070998 (529h25m11.506154043s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.637746432 (529h25m11.508829467s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.642795406 (529h25m11.513878462s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.646797187 (529h25m11.517880222s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.648722423 (529h25m11.519805468s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.653115148 (529h25m11.524198183s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.669518742 (529h25m11.540601777s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.673846613 (529h25m11.544929658s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.682323394 (529h25m11.553406429s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.939294221 (529h25m11.810377266s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.959324092 (529h25m11.830407127s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.042123345 (529h25m11.91320639s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409785817 (529h25m12.280868883s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409842215 (529h25m12.28092526s) [seg_pos={0 0}, upload=464328]: Segment reached optimal threshold of 13 pieces. Dec 27 16:54:12.410069919 (529h25m12.281152974s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410079758 (529h25m12.281162803s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410393385 (529h25m12.28147643s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410409776 (529h25m12.281492821s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410432419 (529h25m12.281515464s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410449933 (529h25m12.281532968s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410468418 (529h25m12.281551463s) [seg_pos={0 0}, upload=464328]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:54:12.410589569 (529h25m12.281672654s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:54:12.410597073 (529h25m12.281680108s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:54:12.695997364 (529h25m12.567080409s) [upload=464328]: Done uploading segments... Dec 27 16:54:12.696056086 (529h25m12.567139121s) [upload=464328]: Upload finished. err: <nil> Duration: 30.002803957s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....X.....X...X.............................X..............X] duration: 5.001182101s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 16:54:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001199825s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......] duration: 5.638203859s trace: https://jaeger.dev.storj.io/trace/38b2c0eda6599275 FAIL (2025-12-27 16:53:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2 (529h19m35.176544717s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:48:35.552842859 (529h19m35.423925924s) [seg_pos={0 0}, upload=464246]: Begin upload segment... Dec 27 16:48:35.552922000 (529h19m35.424005045s) [seg_pos={0 0}, upload=464246]: Done begin upload segment. Dec 27 16:48:35.552987324 (529h19m35.424070359s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553008293 (529h19m35.424091349s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553115347 (529h19m35.424198392s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553538494 (529h19m35.424621529s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554072661 (529h19m35.425155716s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554657194 (529h19m35.425740229s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555114975 (529h19m35.426198021s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555559492 (529h19m35.426642537s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555995812 (529h19m35.427078847s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556000682 (529h19m35.427083717s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556487669 (529h19m35.427570714s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.557470671 (529h19m35.428553747s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558028353 (529h19m35.429111388s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558483550 (529h19m35.429566585s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558976088 (529h19m35.430059123s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.559312778 (529h19m35.430395813s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:37.204335706 (529h19m37.075418741s) [upload=464246]: Next returned nil segment Dec 27 16:48:37.204348490 (529h19m37.075431525s) [upload=464246]: Waiting for error group managing segments... Dec 27 16:48:37.551272132 (529h19m37.422355177s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.553218739 (529h19m37.424301774s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.557494771 (529h19m37.428577816s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.558734783 (529h19m37.429817818s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.568754311 (529h19m37.439837346s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.576611212 (529h19m37.447694247s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583297023 (529h19m37.454380058s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583559943 (529h19m37.454642978s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.584769557 (529h19m37.455852592s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.777830628 (529h19m37.648913663s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.819550982 (529h19m37.690634017s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.846663929 (529h19m37.717746965s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880153289 (529h19m37.751236334s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880174018 (529h19m37.751257063s) [seg_pos={0 0}, upload=464246]: Segment reached optimal threshold of 13 pieces. Dec 27 16:48:37.880371554 (529h19m37.751454589s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection Dec 27 16:48:37.880379900 (529h19m37.751462945s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880451807 (529h19m37.751534852s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection Dec 27 16:48:37.880462527 (529h19m37.751545562s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880733714 (529h19m37.751816769s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection Dec 27 16:48:37.880743633 (529h19m37.751826668s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880765895 (529h19m37.75184893s) [seg_pos={0 0}, upload=464246]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:48:37.880886976 (529h19m37.751970021s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:48:37.880894320 (529h19m37.751977365s) [upload=464246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:48:38.181913271 (529h19m38.052996307s) [upload=464246]: Done uploading segments... Dec 27 16:48:38.181933840 (529h19m38.053016875s) [upload=464246]: Upload finished. err: <nil> Duration:* 5.638233496s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X.....X...X.............................X..............] duration: 3.626582487s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:52:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.626606312s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X.....X...X.............................X.............] duration: 3.5954789s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:52:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.595510971s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX.....] duration: 18.919885425s trace: https://jaeger.dev.storj.io/trace/38b2c0eda6599275 FAIL (2025-12-27 16:52:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2 (529h19m35.176544717s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:48:35.552842859 (529h19m35.423925924s) [seg_pos={0 0}, upload=464246]: Begin upload segment... Dec 27 16:48:35.552922000 (529h19m35.424005045s) [seg_pos={0 0}, upload=464246]: Done begin upload segment. Dec 27 16:48:35.552987324 (529h19m35.424070359s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553008293 (529h19m35.424091349s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553115347 (529h19m35.424198392s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553538494 (529h19m35.424621529s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554072661 (529h19m35.425155716s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554657194 (529h19m35.425740229s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555114975 (529h19m35.426198021s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555559492 (529h19m35.426642537s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555995812 (529h19m35.427078847s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556000682 (529h19m35.427083717s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556487669 (529h19m35.427570714s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.557470671 (529h19m35.428553747s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558028353 (529h19m35.429111388s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558483550 (529h19m35.429566585s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558976088 (529h19m35.430059123s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.559312778 (529h19m35.430395813s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:37.204335706 (529h19m37.075418741s) [upload=464246]: Next returned nil segment Dec 27 16:48:37.204348490 (529h19m37.075431525s) [upload=464246]: Waiting for error group managing segments... Dec 27 16:48:37.551272132 (529h19m37.422355177s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.553218739 (529h19m37.424301774s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.557494771 (529h19m37.428577816s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.558734783 (529h19m37.429817818s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.568754311 (529h19m37.439837346s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.576611212 (529h19m37.447694247s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583297023 (529h19m37.454380058s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583559943 (529h19m37.454642978s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.584769557 (529h19m37.455852592s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.777830628 (529h19m37.648913663s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.819550982 (529h19m37.690634017s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.846663929 (529h19m37.717746965s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880153289 (529h19m37.751236334s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880174018 (529h19m37.751257063s) [seg_pos={0 0}, upload=464246]: Segment reached optimal threshold of 13 pieces. Dec 27 16:48:37.880371554 (529h19m37.751454589s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection Dec 27 16:48:37.880379900 (529h19m37.751462945s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880451807 (529h19m37.751534852s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection Dec 27 16:48:37.880462527 (529h19m37.751545562s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880733714 (529h19m37.751816769s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection Dec 27 16:48:37.880743633 (529h19m37.751826668s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880765895 (529h19m37.75184893s) [seg_pos={0 0}, upload=464246]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:48:37.880886976 (529h19m37.751970021s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:48:37.880894320 (529h19m37.751977365s) [upload=464246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:48:38.181913271 (529h19m38.052996307s) [upload=464246]: Done uploading segments... Dec 27 16:48:38.181933840 (529h19m38.053016875s) [upload=464246]: Upload finished. err: <nil> Duration:* 18.919928166s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X.....X...X.............................X............] duration: 3.533893611s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:52:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533908187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX....] duration: 5.868395102s trace: https://jaeger.dev.storj.io/trace/38b2c0eda6599275 FAIL (2025-12-27 16:51:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2 (529h19m35.176544717s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:48:35.552842859 (529h19m35.423925924s) [seg_pos={0 0}, upload=464246]: Begin upload segment... Dec 27 16:48:35.552922000 (529h19m35.424005045s) [seg_pos={0 0}, upload=464246]: Done begin upload segment. Dec 27 16:48:35.552987324 (529h19m35.424070359s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553008293 (529h19m35.424091349s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553115347 (529h19m35.424198392s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553538494 (529h19m35.424621529s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554072661 (529h19m35.425155716s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554657194 (529h19m35.425740229s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555114975 (529h19m35.426198021s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555559492 (529h19m35.426642537s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555995812 (529h19m35.427078847s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556000682 (529h19m35.427083717s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556487669 (529h19m35.427570714s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.557470671 (529h19m35.428553747s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558028353 (529h19m35.429111388s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558483550 (529h19m35.429566585s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558976088 (529h19m35.430059123s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.559312778 (529h19m35.430395813s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:37.204335706 (529h19m37.075418741s) [upload=464246]: Next returned nil segment Dec 27 16:48:37.204348490 (529h19m37.075431525s) [upload=464246]: Waiting for error group managing segments... Dec 27 16:48:37.551272132 (529h19m37.422355177s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.553218739 (529h19m37.424301774s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.557494771 (529h19m37.428577816s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.558734783 (529h19m37.429817818s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.568754311 (529h19m37.439837346s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.576611212 (529h19m37.447694247s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583297023 (529h19m37.454380058s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583559943 (529h19m37.454642978s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.584769557 (529h19m37.455852592s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.777830628 (529h19m37.648913663s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.819550982 (529h19m37.690634017s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.846663929 (529h19m37.717746965s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880153289 (529h19m37.751236334s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880174018 (529h19m37.751257063s) [seg_pos={0 0}, upload=464246]: Segment reached optimal threshold of 13 pieces. Dec 27 16:48:37.880371554 (529h19m37.751454589s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection Dec 27 16:48:37.880379900 (529h19m37.751462945s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880451807 (529h19m37.751534852s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection Dec 27 16:48:37.880462527 (529h19m37.751545562s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880733714 (529h19m37.751816769s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection Dec 27 16:48:37.880743633 (529h19m37.751826668s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880765895 (529h19m37.75184893s) [seg_pos={0 0}, upload=464246]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:48:37.880886976 (529h19m37.751970021s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:48:37.880894320 (529h19m37.751977365s) [upload=464246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:48:38.181913271 (529h19m38.052996307s) [upload=464246]: Done uploading segments... Dec 27 16:48:38.181933840 (529h19m38.053016875s) [upload=464246]: Upload finished. err: <nil> Duration:* 5.868422503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX...] duration: 5.680193605s trace: https://jaeger.dev.storj.io/trace/38b2c0eda6599275 FAIL (2025-12-27 16:51:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2 (529h19m35.176544717s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:48:35.552842859 (529h19m35.423925924s) [seg_pos={0 0}, upload=464246]: Begin upload segment... Dec 27 16:48:35.552922000 (529h19m35.424005045s) [seg_pos={0 0}, upload=464246]: Done begin upload segment. Dec 27 16:48:35.552987324 (529h19m35.424070359s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553008293 (529h19m35.424091349s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553115347 (529h19m35.424198392s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553538494 (529h19m35.424621529s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554072661 (529h19m35.425155716s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554657194 (529h19m35.425740229s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555114975 (529h19m35.426198021s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555559492 (529h19m35.426642537s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555995812 (529h19m35.427078847s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556000682 (529h19m35.427083717s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556487669 (529h19m35.427570714s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.557470671 (529h19m35.428553747s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558028353 (529h19m35.429111388s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558483550 (529h19m35.429566585s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558976088 (529h19m35.430059123s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.559312778 (529h19m35.430395813s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:37.204335706 (529h19m37.075418741s) [upload=464246]: Next returned nil segment Dec 27 16:48:37.204348490 (529h19m37.075431525s) [upload=464246]: Waiting for error group managing segments... Dec 27 16:48:37.551272132 (529h19m37.422355177s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.553218739 (529h19m37.424301774s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.557494771 (529h19m37.428577816s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.558734783 (529h19m37.429817818s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.568754311 (529h19m37.439837346s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.576611212 (529h19m37.447694247s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583297023 (529h19m37.454380058s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583559943 (529h19m37.454642978s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.584769557 (529h19m37.455852592s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.777830628 (529h19m37.648913663s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.819550982 (529h19m37.690634017s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.846663929 (529h19m37.717746965s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880153289 (529h19m37.751236334s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880174018 (529h19m37.751257063s) [seg_pos={0 0}, upload=464246]: Segment reached optimal threshold of 13 pieces. Dec 27 16:48:37.880371554 (529h19m37.751454589s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection Dec 27 16:48:37.880379900 (529h19m37.751462945s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880451807 (529h19m37.751534852s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection Dec 27 16:48:37.880462527 (529h19m37.751545562s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880733714 (529h19m37.751816769s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection Dec 27 16:48:37.880743633 (529h19m37.751826668s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880765895 (529h19m37.75184893s) [seg_pos={0 0}, upload=464246]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:48:37.880886976 (529h19m37.751970021s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:48:37.880894320 (529h19m37.751977365s) [upload=464246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:48:38.181913271 (529h19m38.052996307s) [upload=464246]: Done uploading segments... Dec 27 16:48:38.181933840 (529h19m38.053016875s) [upload=464246]: Upload finished. err: <nil> Duration:* 5.680291321s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X.....X...X.............................X...........] duration: 3.903268655s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:51:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.903346093s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX..] duration: 5.875002044s trace: https://jaeger.dev.storj.io/trace/38b2c0eda6599275 FAIL (2025-12-27 16:50:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2 (529h19m35.176544717s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:48:35.552842859 (529h19m35.423925924s) [seg_pos={0 0}, upload=464246]: Begin upload segment... Dec 27 16:48:35.552922000 (529h19m35.424005045s) [seg_pos={0 0}, upload=464246]: Done begin upload segment. Dec 27 16:48:35.552987324 (529h19m35.424070359s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553008293 (529h19m35.424091349s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553115347 (529h19m35.424198392s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553538494 (529h19m35.424621529s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554072661 (529h19m35.425155716s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554657194 (529h19m35.425740229s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555114975 (529h19m35.426198021s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555559492 (529h19m35.426642537s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555995812 (529h19m35.427078847s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556000682 (529h19m35.427083717s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556487669 (529h19m35.427570714s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.557470671 (529h19m35.428553747s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558028353 (529h19m35.429111388s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558483550 (529h19m35.429566585s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558976088 (529h19m35.430059123s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.559312778 (529h19m35.430395813s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:37.204335706 (529h19m37.075418741s) [upload=464246]: Next returned nil segment Dec 27 16:48:37.204348490 (529h19m37.075431525s) [upload=464246]: Waiting for error group managing segments... Dec 27 16:48:37.551272132 (529h19m37.422355177s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.553218739 (529h19m37.424301774s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.557494771 (529h19m37.428577816s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.558734783 (529h19m37.429817818s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.568754311 (529h19m37.439837346s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.576611212 (529h19m37.447694247s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583297023 (529h19m37.454380058s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583559943 (529h19m37.454642978s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.584769557 (529h19m37.455852592s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.777830628 (529h19m37.648913663s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.819550982 (529h19m37.690634017s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.846663929 (529h19m37.717746965s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880153289 (529h19m37.751236334s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880174018 (529h19m37.751257063s) [seg_pos={0 0}, upload=464246]: Segment reached optimal threshold of 13 pieces. Dec 27 16:48:37.880371554 (529h19m37.751454589s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection Dec 27 16:48:37.880379900 (529h19m37.751462945s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880451807 (529h19m37.751534852s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection Dec 27 16:48:37.880462527 (529h19m37.751545562s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880733714 (529h19m37.751816769s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection Dec 27 16:48:37.880743633 (529h19m37.751826668s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880765895 (529h19m37.75184893s) [seg_pos={0 0}, upload=464246]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:48:37.880886976 (529h19m37.751970021s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:48:37.880894320 (529h19m37.751977365s) [upload=464246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:48:38.181913271 (529h19m38.052996307s) [upload=464246]: Done uploading segments... Dec 27 16:48:38.181933840 (529h19m38.053016875s) [upload=464246]: Upload finished. err: <nil> Duration:* 5.875024227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........X.....X...X.............................X..........] duration: 4.134655836s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:50:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.134707745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX.] duration: 5.851599504s trace: https://jaeger.dev.storj.io/trace/38b2c0eda6599275 FAIL (2025-12-27 16:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2 (529h19m35.176544717s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:48:35.552842859 (529h19m35.423925924s) [seg_pos={0 0}, upload=464246]: Begin upload segment... Dec 27 16:48:35.552922000 (529h19m35.424005045s) [seg_pos={0 0}, upload=464246]: Done begin upload segment. Dec 27 16:48:35.552987324 (529h19m35.424070359s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553008293 (529h19m35.424091349s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553115347 (529h19m35.424198392s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553538494 (529h19m35.424621529s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554072661 (529h19m35.425155716s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554657194 (529h19m35.425740229s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555114975 (529h19m35.426198021s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555559492 (529h19m35.426642537s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555995812 (529h19m35.427078847s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556000682 (529h19m35.427083717s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556487669 (529h19m35.427570714s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.557470671 (529h19m35.428553747s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558028353 (529h19m35.429111388s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558483550 (529h19m35.429566585s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558976088 (529h19m35.430059123s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.559312778 (529h19m35.430395813s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:37.204335706 (529h19m37.075418741s) [upload=464246]: Next returned nil segment Dec 27 16:48:37.204348490 (529h19m37.075431525s) [upload=464246]: Waiting for error group managing segments... Dec 27 16:48:37.551272132 (529h19m37.422355177s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.553218739 (529h19m37.424301774s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.557494771 (529h19m37.428577816s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.558734783 (529h19m37.429817818s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.568754311 (529h19m37.439837346s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.576611212 (529h19m37.447694247s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583297023 (529h19m37.454380058s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583559943 (529h19m37.454642978s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.584769557 (529h19m37.455852592s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.777830628 (529h19m37.648913663s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.819550982 (529h19m37.690634017s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.846663929 (529h19m37.717746965s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880153289 (529h19m37.751236334s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880174018 (529h19m37.751257063s) [seg_pos={0 0}, upload=464246]: Segment reached optimal threshold of 13 pieces. Dec 27 16:48:37.880371554 (529h19m37.751454589s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection Dec 27 16:48:37.880379900 (529h19m37.751462945s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880451807 (529h19m37.751534852s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection Dec 27 16:48:37.880462527 (529h19m37.751545562s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880733714 (529h19m37.751816769s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection Dec 27 16:48:37.880743633 (529h19m37.751826668s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880765895 (529h19m37.75184893s) [seg_pos={0 0}, upload=464246]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:48:37.880886976 (529h19m37.751970021s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:48:37.880894320 (529h19m37.751977365s) [upload=464246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:48:38.181913271 (529h19m38.052996307s) [upload=464246]: Done uploading segments... Dec 27 16:48:38.181933840 (529h19m38.053016875s) [upload=464246]: Upload finished. err: <nil> Duration:* 5.851642564s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........X.....X...X.............................X.........] duration: 3.722614079s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:49:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.72264124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX] duration: 30.003950348s trace: https://jaeger.dev.storj.io/trace/38b2c0eda6599275 FAIL (2025-12-27 16:49:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2 (529h19m35.176544717s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:48:35.552842859 (529h19m35.423925924s) [seg_pos={0 0}, upload=464246]: Begin upload segment... Dec 27 16:48:35.552922000 (529h19m35.424005045s) [seg_pos={0 0}, upload=464246]: Done begin upload segment. Dec 27 16:48:35.552987324 (529h19m35.424070359s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553008293 (529h19m35.424091349s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553115347 (529h19m35.424198392s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.553538494 (529h19m35.424621529s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554072661 (529h19m35.425155716s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.554657194 (529h19m35.425740229s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555114975 (529h19m35.426198021s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555559492 (529h19m35.426642537s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.555995812 (529h19m35.427078847s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556000682 (529h19m35.427083717s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.556487669 (529h19m35.427570714s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.557470671 (529h19m35.428553747s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558028353 (529h19m35.429111388s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558483550 (529h19m35.429566585s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.558976088 (529h19m35.430059123s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:35.559312778 (529h19m35.430395813s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Uploading piece... Dec 27 16:48:37.204335706 (529h19m37.075418741s) [upload=464246]: Next returned nil segment Dec 27 16:48:37.204348490 (529h19m37.075431525s) [upload=464246]: Waiting for error group managing segments... Dec 27 16:48:37.551272132 (529h19m37.422355177s) [noise=true , address=145.79.227.38:4977 , piece_id=OG7NYITRP3RLJZW56AEJZXSVAPC2MSIEHM6N5FXIZRJ3HTM2S4OA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.553218739 (529h19m37.424301774s) [noise=true , address=145.79.227.48:4303 , piece_id=KJ36JC7FP62L6GTDODGG4SZXTOBLCBM4GCOF3T57ENACVAUMRAMQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.557494771 (529h19m37.428577816s) [noise=true , address=145.79.227.46:4603 , piece_id=QJJV6INJQBD73HDR6TQJJJ7BMK3I73AGREHOSUYGYI2WS73C244Q, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.558734783 (529h19m37.429817818s) [noise=true , address=145.79.227.21:4474 , piece_id=OCAJBR2DSDP4TNRO4JD26STEK4Z22UAGJU5YXJS5IF4OIMHGFC2A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.568754311 (529h19m37.439837346s) [noise=true , address=145.79.227.37:4565 , piece_id=RTM4OPAU3PIM66UIIKJ4H3STBRNVNR6GM7SLJXGGTHCYFMDOOOBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.576611212 (529h19m37.447694247s) [noise=true , address=145.79.227.51:4410 , piece_id=ZD6TG4RKOEHSGV6CIGKILA6OICTCO7C4ACI5RUEBDQ62ORRZMSBA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583297023 (529h19m37.454380058s) [noise=true , address=145.79.227.32:4834 , piece_id=U32SWVLVABNMWFMUYJSXLDJMXGWD7JJTE5OQWLI7SQDKCF3TPTXA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.583559943 (529h19m37.454642978s) [noise=true , address=145.79.227.45:4230 , piece_id=XRUB5P4KCAGWCP4ASRFEQCYOQTI7FHTIO6LBNASFF4D4YCB7ERMA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.584769557 (529h19m37.455852592s) [noise=true , address=145.79.227.52:4445 , piece_id=PLAT7EPWB7HXP44EIX3P25IBH7UO4RNOQYH2AXU5IFX6W5FL6BPA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.777830628 (529h19m37.648913663s) [noise=true , address=77.90.31.17:4770 , piece_id=5CQCGCUMKMEXGTFCDA5ZZMGMPES5R6SRYNOKHIPTBKDYP3OH5XTA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.819550982 (529h19m37.690634017s) [noise=true , address=45.91.116.9:4263 , piece_id=QPFFTTAS2FNMQ2M6U2MMNEA5VA4S5ADGJA76TB2IZHZFSFH73XYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.846663929 (529h19m37.717746965s) [noise=true , address=45.91.116.13:4154 , piece_id=URY3ENYP6HGWLBWRYTOGQKSQ3DOUZT2I7WSZJPBUAMAXAPW4PHIQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880153289 (529h19m37.751236334s) [noise=true , address=77.90.31.21:4877 , piece_id=BGPEQYPWPF3HFPMDC7PNYIUHTKRZGEMSMOXYMYQERR7GKONBXGYA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: <nil> Dec 27 16:48:37.880174018 (529h19m37.751257063s) [seg_pos={0 0}, upload=464246]: Segment reached optimal threshold of 13 pieces. Dec 27 16:48:37.880371554 (529h19m37.751454589s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47872->45.91.116.4:4940: use of closed network connection Dec 27 16:48:37.880379900 (529h19m37.751462945s) [noise=true , address=45.91.116.4:4940 , piece_id=WPJYO36NHRNE66GRAF3OV6LXKZMQSD6FPFKF6HI767FKXSQ3G7LA, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880451807 (529h19m37.751534852s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41042->145.79.227.4:4187: use of closed network connection Dec 27 16:48:37.880462527 (529h19m37.751545562s) [noise=true , address=145.79.227.4:4187 , piece_id=5X2AZIJXG6PYXXATSFWFAFJYK52Y3WQXBXAZJKIUIKD36SMMPRHQ, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880733714 (529h19m37.751816769s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48552->45.91.116.12:4225: use of closed network connection Dec 27 16:48:37.880743633 (529h19m37.751826668s) [noise=true , address=45.91.116.12:4225 , piece_id=7HWVW52EYCAEFVDLV77M6KVX6QMRFUAEPSOAG3A6HZKZUPSFYM7A, seg_pos={0 0}, upload=464246]: Long tail cancellation due to optimal threshold Dec 27 16:48:37.880765895 (529h19m37.75184893s) [seg_pos={0 0}, upload=464246]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:48:37.880886976 (529h19m37.751970021s) [upload=464246]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:48:37.880894320 (529h19m37.751977365s) [upload=464246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:48:38.181913271 (529h19m38.052996307s) [upload=464246]: Done uploading segments... Dec 27 16:48:38.181933840 (529h19m38.053016875s) [upload=464246]: Upload finished. err: <nil> Duration:* 30.003976177s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X.....X...X.............................X........] duration: 3.806219086s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:48:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.80628949s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X.....X...X.............................X.......] duration: 3.587476603s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:48:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.587507291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XX] duration: 30.003921644s trace: https://jaeger.dev.storj.io/trace/31936289a50dd333 FAIL (2025-12-27 16:48:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: UX6UCPZZLFNSCAOSA7HKAXSOPYJLMV7BGDOYKVDRHP4OOWUWPS2A) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 75ZNUEXAR4FGJBS36A45YG3IDXY3XTFGTFCXQ7I3PHC2BU7Z75MQ) context deadline exceeded ======LOGS====== Dec 27 16:47:46.906586343 (529h18m46.777669378s) [upload=464238]: Starting upload Dec 27 16:47:46.906868961 (529h18m46.777952007s) [upload=464238]: Done starting upload Dec 27 16:47:46.907113507 (529h18m46.778196552s) [upload=464238]: Uploading segments... Dec 27 16:47:46.907153844 (529h18m46.778236879s) [upload=464238]: Got next segment. Inline: false Dec 27 16:47:46.907215091 (529h18m46.778298136s) [upload=464238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:47:46.907222004 (529h18m46.778305049s) [upload=464238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:47:47.150318923 (529h18m47.021401958s) [seg_pos={0 0}, upload=464238]: Begin upload segment... Dec 27 16:47:47.150405146 (529h18m47.021488181s) [seg_pos={0 0}, upload=464238]: Done begin upload segment. Dec 27 16:47:47.150459861 (529h18m47.021542926s) [noise=true , address=145.79.227.45:4230 , piece_id=HD5R4DSSM7HG6IMGUYSWXEU5RHVLCXIPKEGFXSY2Z6GZPOBKFNYQ, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.150482043 (529h18m47.021565078s) [noise=true , address=45.91.116.13:4153 , piece_id=6IJTPGFUK4EKPEOL2HEP3F42VVJ472GEGM3PUC2ETSGZ3KUJSI4A, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.151317744 (529h18m47.022400789s) [noise=true , address=145.79.227.40:4051 , piece_id=ZL6XMUT3OEDWZROJB5NHWAJCHLPT3HICD3SYBAUQMMERYBLK3CXQ, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.151998711 (529h18m47.023081756s) [noise=true , address=145.79.227.32:4834 , piece_id=SOECQ7D52DP65DLCTWB5HGCFSATUQP3MO5K6Y6ASYTBFQ6GZ47AA, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.152590247 (529h18m47.023673292s) [noise=true , address=145.79.227.10:4403 , piece_id=SHVSKIAZL646KNZHJ5GBBCQUWSX4IKZBRCJY2DC2WL2VSEJ3VRCA, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.153068007 (529h18m47.024151042s) [noise=true , address=45.91.116.10:4082 , piece_id=HMIPZZP5ZJQMX6BEPSBVMOKTWVSRI2QO6IKZ27JWKIWGEB4AW3BQ, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.153106651 (529h18m47.024189696s) [noise=true , address=77.90.31.10:4518 , piece_id=75ZNUEXAR4FGJBS36A45YG3IDXY3XTFGTFCXQ7I3PHC2BU7Z75MQ, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.153578739 (529h18m47.024661794s) [noise=true , address=145.79.227.48:4302 , piece_id=ES37P7UMFASYHOXFHHKSA3446TXBCZKJETDQAEH7HOSPVBCERSJQ, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.153789100 (529h18m47.024872145s) [noise=true , address=77.90.31.19:4842 , piece_id=R3NN3X2SIBD6DWJIMOEVBA337KHVQF7FC7ZUXKWWEVPCXQA7HF3A, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.154075326 (529h18m47.025158371s) [noise=true , address=45.91.116.4:4939 , piece_id=CJDTJSVL6YGTR4WYEN2KZAQE2KKS4D6GA2WEZNGQULLC6U656K7A, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.154275346 (529h18m47.025358381s) [noise=true , address=145.79.227.14:4839 , piece_id=WFG5METJ37KHC4KFZGMCX37TNPI6A6HAFK2ERZORAAOJ2UAUK5UA, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.154670449 (529h18m47.025753494s) [noise=true , address=145.79.227.37:4567 , piece_id=F2OX3HO6RSL2V7X2Y52C4ODNJSWJULQ2QEHN5CP4VWBYNVPHVIDA, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.155176963 (529h18m47.026259998s) [noise=true , address=77.90.31.21:4878 , piece_id=UX6UCPZZLFNSCAOSA7HKAXSOPYJLMV7BGDOYKVDRHP4OOWUWPS2A, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.155627320 (529h18m47.026710365s) [noise=true , address=77.90.31.17:4773 , piece_id=YKODONG2NIJAQ6IKRP32ADH2QYVUMXQVQNZRYSO4DLBJAIAOM74Q, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.156100902 (529h18m47.027183948s) [noise=true , address=145.79.227.9:4365 , piece_id=IMDXR6QPXOI6PPANR2H57FQ2Z4HZWDU5MGNHTU7TBEOBU4HHB73Q, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:47.157276692 (529h18m47.028359737s) [noise=true , address=145.79.227.27:4691 , piece_id=YMTSHVEDGAKZFP2QS3XRV2LOW2PBLIEYQQGYYP3D6ACWDWOFS4WQ, seg_pos={0 0}, upload=464238]: Uploading piece... Dec 27 16:47:48.869242369 (529h18m48.740325414s) [upload=464238]: Next returned nil segment Dec 27 16:47:48.869258218 (529h18m48.740341253s) [upload=464238]: Waiting for error group managing segments... Dec 27 16:47:49.156726287 (529h18m49.027809322s) [noise=true , address=145.79.227.10:4403 , piece_id=SHVSKIAZL646KNZHJ5GBBCQUWSX4IKZBRCJY2DC2WL2VSEJ3VRCA, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.158058423 (529h18m49.029141458s) [noise=true , address=145.79.227.45:4230 , piece_id=HD5R4DSSM7HG6IMGUYSWXEU5RHVLCXIPKEGFXSY2Z6GZPOBKFNYQ, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.160001684 (529h18m49.031084719s) [noise=true , address=145.79.227.14:4839 , piece_id=WFG5METJ37KHC4KFZGMCX37TNPI6A6HAFK2ERZORAAOJ2UAUK5UA, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.160342122 (529h18m49.031425167s) [noise=true , address=145.79.227.27:4691 , piece_id=YMTSHVEDGAKZFP2QS3XRV2LOW2PBLIEYQQGYYP3D6ACWDWOFS4WQ, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.177275406 (529h18m49.048358441s) [noise=true , address=145.79.227.48:4302 , piece_id=ES37P7UMFASYHOXFHHKSA3446TXBCZKJETDQAEH7HOSPVBCERSJQ, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.188025716 (529h18m49.059108752s) [noise=true , address=145.79.227.32:4834 , piece_id=SOECQ7D52DP65DLCTWB5HGCFSATUQP3MO5K6Y6ASYTBFQ6GZ47AA, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.197591141 (529h18m49.068674176s) [noise=true , address=145.79.227.9:4365 , piece_id=IMDXR6QPXOI6PPANR2H57FQ2Z4HZWDU5MGNHTU7TBEOBU4HHB73Q, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.257430565 (529h18m49.128513601s) [noise=true , address=145.79.227.40:4051 , piece_id=ZL6XMUT3OEDWZROJB5NHWAJCHLPT3HICD3SYBAUQMMERYBLK3CXQ, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.265441910 (529h18m49.136524955s) [noise=true , address=145.79.227.37:4567 , piece_id=F2OX3HO6RSL2V7X2Y52C4ODNJSWJULQ2QEHN5CP4VWBYNVPHVIDA, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.373389151 (529h18m49.244472196s) [noise=true , address=77.90.31.10:4518 , piece_id=75ZNUEXAR4FGJBS36A45YG3IDXY3XTFGTFCXQ7I3PHC2BU7Z75MQ, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.407192318 (529h18m49.278275353s) [noise=true , address=77.90.31.21:4878 , piece_id=UX6UCPZZLFNSCAOSA7HKAXSOPYJLMV7BGDOYKVDRHP4OOWUWPS2A, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.435235717 (529h18m49.306318762s) [noise=true , address=45.91.116.4:4939 , piece_id=CJDTJSVL6YGTR4WYEN2KZAQE2KKS4D6GA2WEZNGQULLC6U656K7A, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.435334365 (529h18m49.3064174s) [noise=true , address=77.90.31.19:4842 , piece_id=R3NN3X2SIBD6DWJIMOEVBA337KHVQF7FC7ZUXKWWEVPCXQA7HF3A, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.435354463 (529h18m49.306437508s) [seg_pos={0 0}, upload=464238]: Segment reached optimal threshold of 13 pieces. Dec 27 16:47:49.435551418 (529h18m49.306634454s) [noise=true , address=45.91.116.13:4153 , piece_id=6IJTPGFUK4EKPEOL2HEP3F42VVJ472GEGM3PUC2ETSGZ3KUJSI4A, seg_pos={0 0}, upload=464238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 16:47:49.435574122 (529h18m49.306657157s) [noise=true , address=45.91.116.13:4153 , piece_id=6IJTPGFUK4EKPEOL2HEP3F42VVJ472GEGM3PUC2ETSGZ3KUJSI4A, seg_pos={0 0}, upload=464238]: Long tail cancellation due to optimal threshold Dec 27 16:47:49.435706513 (529h18m49.306789558s) [noise=true , address=77.90.31.17:4773 , piece_id=YKODONG2NIJAQ6IKRP32ADH2QYVUMXQVQNZRYSO4DLBJAIAOM74Q, seg_pos={0 0}, upload=464238]: Done uploading piece. err: <nil> Dec 27 16:47:49.435760587 (529h18m49.306843632s) [noise=true , address=45.91.116.10:4082 , piece_id=HMIPZZP5ZJQMX6BEPSBVMOKTWVSRI2QO6IKZ27JWKIWGEB4AW3BQ, seg_pos={0 0}, upload=464238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 16:47:49.435767560 (529h18m49.306850595s) [noise=true , address=45.91.116.10:4082 , piece_id=HMIPZZP5ZJQMX6BEPSBVMOKTWVSRI2QO6IKZ27JWKIWGEB4AW3BQ, seg_pos={0 0}, upload=464238]: Long tail cancellation due to optimal threshold Dec 27 16:47:49.435784322 (529h18m49.306867357s) [seg_pos={0 0}, upload=464238]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 27 16:47:49.435947182 (529h18m49.307030227s) [upload=464238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:47:49.435955958 (529h18m49.307038993s) [upload=464238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:47:49.718446910 (529h18m49.589529945s) [upload=464238]: Done uploading segments... Dec 27 16:47:49.718471115 (529h18m49.58955416s) [upload=464238]: Upload finished. err: <nil> Duration: 30.003935982s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X.....X...X.............................X......] duration: 3.518263258s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:48:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.518288195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.X] duration: 30.003152693s trace: https://jaeger.dev.storj.io/trace/3ee29d3322e89438 FAIL (2025-12-27 16:47:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: IUANTBLHMHFLOVKBNMYGOB6SA5ME6WS7O4MYUIMLJC46H2VFKFQA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 3Z7L4P2QB4GKSINMRBJD3ZDUTH3MHJRSYXNTWSWGMEWXQAOBDRWA) context deadline exceeded ======LOGS====== 16:47:16.643731593 (529h18m16.514814639s) [upload=464231]: Starting upload Dec 27 16:47:16.643827576 (529h18m16.514910612s) [upload=464231]: Done starting upload Dec 27 16:47:16.643887991 (529h18m16.514971026s) [upload=464231]: Uploading segments... Dec 27 16:47:16.644000175 (529h18m16.51508321s) [upload=464231]: Got next segment. Inline: false Dec 27 16:47:16.644030843 (529h18m16.515113878s) [upload=464231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:47:16.644035122 (529h18m16.515118157s) [upload=464231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:47:16.890402267 (529h18m16.761485302s) [seg_pos={0 0}, upload=464231]: Begin upload segment... Dec 27 16:47:16.890445961 (529h18m16.761528996s) [seg_pos={0 0}, upload=464231]: Done begin upload segment. Dec 27 16:47:16.890482631 (529h18m16.761565666s) [noise=true , address=45.91.116.10:4082 , piece_id=GOR6AX76DZEGKC5FKAUETZTYBPTGGQDCTFFJPHDOYHKJIN3BRJKQ, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.890580888 (529h18m16.761663923s) [noise=true , address=145.79.227.35:4481 , piece_id=3BAB5BH4UOWGMJATIPM2SJN5CWWFVNYPSMPDFSWQP5NXQGI5ZIDA, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.891245012 (529h18m16.762328058s) [noise=true , address=145.79.227.51:4409 , piece_id=4LX52M256SUACKFIKIIUFQ35Z3JNTKTTY6FO3JVLJCVQCZZZO6OQ, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.891826699 (529h18m16.762909734s) [noise=true , address=45.91.116.14:4191 , piece_id=3T6TBAMG2TNWLCRS22O5F4TUSOQ4JFHMNBOABTNKXM3TGGXPPGOA, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.892241469 (529h18m16.763324514s) [noise=true , address=145.79.227.47:4266 , piece_id=Z6ORRLAUSY6SG6SWI4IJBPNW472FE6QWGG4NGCZ6XPPUPMUOEH3A, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.892573722 (529h18m16.763656767s) [noise=true , address=145.79.227.28:4725 , piece_id=UEMVT74LAJDA6EUDQZK3OQGPVBPSBUDEM5LSPBMPYXAZQE5ANLRQ, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.892901947 (529h18m16.763984992s) [noise=true , address=145.79.227.50:4373 , piece_id=KPWOFTL5FDJ32JAZDERC7LHCFFI7NTZPCBRH72HIS75LX3KRMENA, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.893963459 (529h18m16.765046494s) [noise=true , address=45.91.116.5:4300 , piece_id=IUANTBLHMHFLOVKBNMYGOB6SA5ME6WS7O4MYUIMLJC46H2VFKFQA, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.894378279 (529h18m16.765461314s) [noise=true , address=145.79.227.36:4518 , piece_id=AZDPJT7SRPCYIIDDY7CKLPD26ZAE5QULHZ2WTRAE33H6FRTUOP6A, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.895588012 (529h18m16.766671057s) [noise=true , address=145.79.227.37:4566 , piece_id=GHLIQ34SCQIRL2GSPG5GROFV3C44JDWKCBQO3DODAUAGGI4DMENA, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.896355553 (529h18m16.767438588s) [noise=true , address=145.79.227.18:4007 , piece_id=6ACMCJWTEHNIYD7HG5KVGZTIWQ2X3XTGT2CEEP5ARATVMSXQ577A, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.896997907 (529h18m16.768080942s) [noise=true , address=145.79.227.41:4085 , piece_id=N73VLKTZHWRWGGZ3EFJFHMPYEH53ILVGGNWZWO47DIFQJEU2AQOQ, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.898195316 (529h18m16.769278351s) [noise=true , address=45.91.116.8:4047 , piece_id=3Z7L4P2QB4GKSINMRBJD3ZDUTH3MHJRSYXNTWSWGMEWXQAOBDRWA, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.899318145 (529h18m16.77040118s) [noise=true , address=145.79.227.22:4509 , piece_id=Z5ORDXLCGR2UXKODGJSDLZUKPMLIE2RYLA6NTJTUW3WXSQ77HHHQ, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.900406057 (529h18m16.771489102s) [noise=true , address=77.90.31.13:4629 , piece_id=PSRTMKIBAQ5GYJAOS4RKR4BUQFX6U2IBGIMFF65BEJY6WTSEPBDQ, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:16.901729727 (529h18m16.772812772s) [noise=true , address=45.91.116.13:4154 , piece_id=NH3DDLEZGOVVL5JKOALU3S5PKHR4DW7PZKYTI3YVQTOG4SVTKX3A, seg_pos={0 0}, upload=464231]: Uploading piece... Dec 27 16:47:18.588812573 (529h18m18.459895608s) [upload=464231]: Next returned nil segment Dec 27 16:47:18.588826139 (529h18m18.459909184s) [upload=464231]: Waiting for error group managing segments... Dec 27 16:47:18.891148583 (529h18m18.762231618s) [noise=true , address=145.79.227.50:4373 , piece_id=KPWOFTL5FDJ32JAZDERC7LHCFFI7NTZPCBRH72HIS75LX3KRMENA, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:18.894906179 (529h18m18.765989224s) [noise=true , address=145.79.227.18:4007 , piece_id=6ACMCJWTEHNIYD7HG5KVGZTIWQ2X3XTGT2CEEP5ARATVMSXQ577A, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:18.899116184 (529h18m18.770199229s) [noise=true , address=145.79.227.51:4409 , piece_id=4LX52M256SUACKFIKIIUFQ35Z3JNTKTTY6FO3JVLJCVQCZZZO6OQ, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:18.899862045 (529h18m18.77094509s) [noise=true , address=145.79.227.37:4566 , piece_id=GHLIQ34SCQIRL2GSPG5GROFV3C44JDWKCBQO3DODAUAGGI4DMENA, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:18.902527330 (529h18m18.773610375s) [noise=true , address=145.79.227.28:4725 , piece_id=UEMVT74LAJDA6EUDQZK3OQGPVBPSBUDEM5LSPBMPYXAZQE5ANLRQ, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:18.910628546 (529h18m18.781711591s) [noise=true , address=145.79.227.47:4266 , piece_id=Z6ORRLAUSY6SG6SWI4IJBPNW472FE6QWGG4NGCZ6XPPUPMUOEH3A, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:18.915860660 (529h18m18.786943705s) [noise=true , address=145.79.227.36:4518 , piece_id=AZDPJT7SRPCYIIDDY7CKLPD26ZAE5QULHZ2WTRAE33H6FRTUOP6A, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:18.933613834 (529h18m18.804696869s) [noise=true , address=145.79.227.41:4085 , piece_id=N73VLKTZHWRWGGZ3EFJFHMPYEH53ILVGGNWZWO47DIFQJEU2AQOQ, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:18.939052029 (529h18m18.810135084s) [noise=true , address=145.79.227.22:4509 , piece_id=Z5ORDXLCGR2UXKODGJSDLZUKPMLIE2RYLA6NTJTUW3WXSQ77HHHQ, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:19.151950204 (529h18m19.023033239s) [noise=true , address=45.91.116.13:4154 , piece_id=NH3DDLEZGOVVL5JKOALU3S5PKHR4DW7PZKYTI3YVQTOG4SVTKX3A, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:19.164879814 (529h18m19.03596286s) [noise=true , address=45.91.116.8:4047 , piece_id=3Z7L4P2QB4GKSINMRBJD3ZDUTH3MHJRSYXNTWSWGMEWXQAOBDRWA, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:19.181506424 (529h18m19.052589469s) [noise=true , address=45.91.116.14:4191 , piece_id=3T6TBAMG2TNWLCRS22O5F4TUSOQ4JFHMNBOABTNKXM3TGGXPPGOA, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:19.876395321 (529h18m19.747478366s) [noise=true , address=45.91.116.5:4300 , piece_id=IUANTBLHMHFLOVKBNMYGOB6SA5ME6WS7O4MYUIMLJC46H2VFKFQA, seg_pos={0 0}, upload=464231]: Done uploading piece. err: <nil> Dec 27 16:47:19.876415270 (529h18m19.747498305s) [seg_pos={0 0}, upload=464231]: Segment reached optimal threshold of 13 pieces. Dec 27 16:47:19.876610280 (529h18m19.747693316s) [noise=true , address=45.91.116.10:4082 , piece_id=GOR6AX76DZEGKC5FKAUETZTYBPTGGQDCTFFJPHDOYHKJIN3BRJKQ, seg_pos={0 0}, upload=464231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39570->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39570->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39570->45.91.116.10:4082: use of closed network connection Dec 27 16:47:19.876615671 (529h18m19.747698706s) [noise=true , address=45.91.116.10:4082 , piece_id=GOR6AX76DZEGKC5FKAUETZTYBPTGGQDCTFFJPHDOYHKJIN3BRJKQ, seg_pos={0 0}, upload=464231]: Long tail cancellation due to optimal threshold Dec 27 16:47:19.876738966 (529h18m19.747822001s) [noise=true , address=77.90.31.13:4629 , piece_id=PSRTMKIBAQ5GYJAOS4RKR4BUQFX6U2IBGIMFF65BEJY6WTSEPBDQ, seg_pos={0 0}, upload=464231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 27 16:47:19.876745668 (529h18m19.747828703s) [noise=true , address=77.90.31.13:4629 , piece_id=PSRTMKIBAQ5GYJAOS4RKR4BUQFX6U2IBGIMFF65BEJY6WTSEPBDQ, seg_pos={0 0}, upload=464231]: Long tail cancellation due to optimal threshold Dec 27 16:47:19.876895914 (529h18m19.747978949s) [noise=true , address=145.79.227.35:4481 , piece_id=3BAB5BH4UOWGMJATIPM2SJN5CWWFVNYPSMPDFSWQP5NXQGI5ZIDA, seg_pos={0 0}, upload=464231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 27 16:47:19.876902097 (529h18m19.747985132s) [noise=true , address=145.79.227.35:4481 , piece_id=3BAB5BH4UOWGMJATIPM2SJN5CWWFVNYPSMPDFSWQP5NXQGI5ZIDA, seg_pos={0 0}, upload=464231]: Long tail cancellation due to optimal threshold Dec 27 16:47:19.876916634 (529h18m19.747999669s) [seg_pos={0 0}, upload=464231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:47:19.876980456 (529h18m19.748063491s) [upload=464231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:47:19.876984724 (529h18m19.748067759s) [upload=464231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:47:20.171359595 (529h18m20.04244264s) [upload=464231]: Done uploading segments... Dec 27 16:47:20.171384502 (529h18m20.042467537s) [upload=464231]: Upload finished. err: <nil> Duration: 30.003174154s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.............X.....X...X.............................X.....] duration: 3.619931893s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:47:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.619957322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X.............X.....X...X.............................X....] duration: 3.857528477s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:46:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.857551471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.] duration: 6.004309247s trace: https://jaeger.dev.storj.io/trace/5f451c9d9242c3f4 FAIL (2025-12-27 16:46:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 16:45:34.069490241 (529h16m33.940573276s) [upload=464207]: Starting upload Dec 27 16:45:34.069553602 (529h16m33.940636637s) [upload=464207]: Done starting upload Dec 27 16:45:34.069734106 (529h16m33.940817141s) [upload=464207]: Uploading segments... Dec 27 16:45:34.069771537 (529h16m33.940854572s) [upload=464207]: Got next segment. Inline: false Dec 27 16:45:34.069823857 (529h16m33.940906902s) [upload=464207]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:45:34.069827965 (529h16m33.940911s) [upload=464207]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:45:34.316020208 (529h16m34.187103243s) [seg_pos={0 0}, upload=464207]: Begin upload segment... Dec 27 16:45:34.316086865 (529h16m34.18716991s) [seg_pos={0 0}, upload=464207]: Done begin upload segment. Dec 27 16:45:34.316114206 (529h16m34.187197241s) [noise=true , address=45.91.116.12:4228 , piece_id=7BB6GZGHQSIVDIV4V337YA2WF2N3VUSPQOLBVLHGHGEGSWU5YO7Q, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.317027095 (529h16m34.18811013s) [noise=true , address=145.79.227.10:4403 , piece_id=SRXUTRXFHAETBGQPAAKX5BYCRREWOKPOAFMWTJJGG3RBORN2JW6Q, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.317554920 (529h16m34.188637955s) [noise=true , address=145.79.227.11:4730 , piece_id=7UYEQXG5U3HNNYK5YGIQ6KDR2OORUAWHNYJ42MG2465QXEGMWWOQ, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.317927930 (529h16m34.189011005s) [noise=true , address=77.90.31.21:4877 , piece_id=NXSR63KJIOFACEV3ZJBIVY7EUTHLT22A2FXACY34TBRKUXQRBIKA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.318436258 (529h16m34.189519293s) [noise=true , address=145.79.227.15:4874 , piece_id=JAFPBFFD3OGMJZLFLRNKDX5SGQC52WPWYS2AGLILHDZ7E5JE5DHA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.318794620 (529h16m34.189877665s) [noise=true , address=145.79.227.51:4409 , piece_id=JH6W4GGR6XEMT4UUVEK7AQOT4MNSAJR4CFOHGBVNLCR3ZIIFE72A, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.320005085 (529h16m34.19108813s) [noise=true , address=45.91.116.14:4189 , piece_id=GQTEMSSDZBBUXFVQYIIFD7XT4NL4FKFDVQW7G3VMG4QMXQHEBBLA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.321276897 (529h16m34.192359933s) [noise=true , address=145.79.227.5:4221 , piece_id=UNP2MGLGIG7ZIRYCVJP5XV6UP5N6244CZMUPEZLSNXVBYGSJHF5Q, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.322798915 (529h16m34.19388195s) [noise=true , address=145.79.227.46:4602 , piece_id=63BRPU4J42AFCAVOXPXX4NT37662XIZXIJ4KO4AIPZWAS2U2XSGA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.324085274 (529h16m34.19516832s) [noise=true , address=77.90.31.14:4661 , piece_id=U6KYKLRNE66MELWOEAZGAKO5PU624WYV7XZYXVIHAQ2ARRLTJTGA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.325188936 (529h16m34.196271981s) [noise=true , address=145.79.227.52:4447 , piece_id=MMAOFUPHPB33H6UD5JKYP4DSXD3DQGWJLXZO3TIJE2AJHHNI3NIQ, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.325678749 (529h16m34.196761784s) [noise=true , address=145.79.227.37:4567 , piece_id=765TDBXSZCMTZC46VT2GS34CG664HQ2VJVMZMDEFLIBNF6VEBZ4A, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.327052555 (529h16m34.19813559s) [noise=true , address=45.91.116.8:4046 , piece_id=IGL3M2LM5BM5ODKJ35WFS2OVCO7IMSB6JVIE5WAYX2TIJGRFRHKQ, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.327567114 (529h16m34.198650149s) [noise=true , address=145.79.227.13:4804 , piece_id=4B2GMMMBDQ6N7P7GQZJHY2EEDMQSHMQM7B2FGVVOXYYWAKQHFDBQ, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.328792789 (529h16m34.199875824s) [noise=true , address=77.90.31.12:4590 , piece_id=2HLPEVPDS3SQI6D27AQMRAHC4PZRCAWHQO37FQHRO2MJW5DQCAQQ, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.329398001 (529h16m34.200481046s) [noise=true , address=45.91.116.13:4153 , piece_id=PXA6VYD4ZGPVOT4Z2OMPPMRJGY6R7T4DGIOJW4K5HK5MRHCF6YFA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:36.038865041 (529h16m35.909948086s) [upload=464207]: Next returned nil segment Dec 27 16:45:36.038877875 (529h16m35.90996091s) [upload=464207]: Waiting for error group managing segments... Dec 27 16:45:36.321180559 (529h16m36.192263594s) [noise=true , address=145.79.227.37:4567 , piece_id=765TDBXSZCMTZC46VT2GS34CG664HQ2VJVMZMDEFLIBNF6VEBZ4A, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.324305990 (529h16m36.195389025s) [noise=true , address=145.79.227.51:4409 , piece_id=JH6W4GGR6XEMT4UUVEK7AQOT4MNSAJR4CFOHGBVNLCR3ZIIFE72A, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.324634515 (529h16m36.19571755s) [noise=true , address=145.79.227.15:4874 , piece_id=JAFPBFFD3OGMJZLFLRNKDX5SGQC52WPWYS2AGLILHDZ7E5JE5DHA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.329053570 (529h16m36.200136605s) [noise=true , address=145.79.227.52:4447 , piece_id=MMAOFUPHPB33H6UD5JKYP4DSXD3DQGWJLXZO3TIJE2AJHHNI3NIQ, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.334638053 (529h16m36.205721098s) [noise=true , address=145.79.227.46:4602 , piece_id=63BRPU4J42AFCAVOXPXX4NT37662XIZXIJ4KO4AIPZWAS2U2XSGA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.336447619 (529h16m36.207530654s) [noise=true , address=145.79.227.11:4730 , piece_id=7UYEQXG5U3HNNYK5YGIQ6KDR2OORUAWHNYJ42MG2465QXEGMWWOQ, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.350231867 (529h16m36.221314902s) [noise=true , address=145.79.227.10:4403 , piece_id=SRXUTRXFHAETBGQPAAKX5BYCRREWOKPOAFMWTJJGG3RBORN2JW6Q, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.357434071 (529h16m36.228517116s) [noise=true , address=145.79.227.13:4804 , piece_id=4B2GMMMBDQ6N7P7GQZJHY2EEDMQSHMQM7B2FGVVOXYYWAKQHFDBQ, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.657384469 (529h16m36.528484396s) [noise=true , address=45.91.116.14:4189 , piece_id=GQTEMSSDZBBUXFVQYIIFD7XT4NL4FKFDVQW7G3VMG4QMXQHEBBLA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.673695106 (529h16m36.544778142s) [noise=true , address=45.91.116.13:4153 , piece_id=PXA6VYD4ZGPVOT4Z2OMPPMRJGY6R7T4DGIOJW4K5HK5MRHCF6YFA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.683037056 (529h16m36.554120101s) [noise=true , address=77.90.31.14:4661 , piece_id=U6KYKLRNE66MELWOEAZGAKO5PU624WYV7XZYXVIHAQ2ARRLTJTGA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.830363700 (529h16m36.701446735s) [noise=true , address=77.90.31.21:4877 , piece_id=NXSR63KJIOFACEV3ZJBIVY7EUTHLT22A2FXACY34TBRKUXQRBIKA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:37.690230205 (529h16m37.56131324s) [noise=true , address=45.91.116.12:4228 , piece_id=7BB6GZGHQSIVDIV4V337YA2WF2N3VUSPQOLBVLHGHGEGSWU5YO7Q, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:37.690247017 (529h16m37.561330052s) [seg_pos={0 0}, upload=464207]: Segment reached optimal threshold of 13 pieces. Dec 27 16:45:37.690530707 (529h16m37.561613742s) [noise=true , address=77.90.31.12:4590 , piece_id=2HLPEVPDS3SQI6D27AQMRAHC4PZRCAWHQO37FQHRO2MJW5DQCAQQ, seg_pos={0 0}, upload=464207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: read tcp 188.166.225.170:53180->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 16:45:37.690538241 (529h16m37.561621276s) [noise=true , address=77.90.31.12:4590 , piece_id=2HLPEVPDS3SQI6D27AQMRAHC4PZRCAWHQO37FQHRO2MJW5DQCAQQ, seg_pos={0 0}, upload=464207]: Long tail cancellation due to optimal threshold Dec 27 16:45:37.690671705 (529h16m37.56175475s) [noise=true , address=45.91.116.8:4046 , piece_id=IGL3M2LM5BM5ODKJ35WFS2OVCO7IMSB6JVIE5WAYX2TIJGRFRHKQ, seg_pos={0 0}, upload=464207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: read tcp 188.166.225.170:57622->45.91.116.8:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 16:45:37.690679890 (529h16m37.561762935s) [noise=true , address=45.91.116.8:4046 , piece_id=IGL3M2LM5BM5ODKJ35WFS2OVCO7IMSB6JVIE5WAYX2TIJGRFRHKQ, seg_pos={0 0}, upload=464207]: Long tail cancellation due to optimal threshold Dec 27 16:45:37.690683959 (529h16m37.561766994s) [noise=true , address=145.79.227.5:4221 , piece_id=UNP2MGLGIG7ZIRYCVJP5XV6UP5N6244CZMUPEZLSNXVBYGSJHF5Q, seg_pos={0 0}, upload=464207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 27 16:45:37.690692174 (529h16m37.56177522s) [noise=true , address=145.79.227.5:4221 , piece_id=UNP2MGLGIG7ZIRYCVJP5XV6UP5N6244CZMUPEZLSNXVBYGSJHF5Q, seg_pos={0 0}, upload=464207]: Long tail cancellation due to optimal threshold Dec 27 16:45:37.690708515 (529h16m37.56179155s) [seg_pos={0 0}, upload=464207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:45:37.690779310 (529h16m37.561862345s) [upload=464207]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:45:37.690789330 (529h16m37.561872365s) [upload=464207]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:45:37.982693866 (529h16m37.853776901s) [upload=464207]: Done uploading segments... Dec 27 16:45:37.982711971 (529h16m37.853795016s) [upload=464207]: Upload finished. err: <nil> Duration: 6.004337341s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X.............X.....X...X.............................X...] duration: 3.472960564s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:46:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.4729852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX] duration: 30.003059107s trace: https://jaeger.dev.storj.io/trace/5f451c9d9242c3f4 FAIL (2025-12-27 16:46:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 16:45:34.069490241 (529h16m33.940573276s) [upload=464207]: Starting upload Dec 27 16:45:34.069553602 (529h16m33.940636637s) [upload=464207]: Done starting upload Dec 27 16:45:34.069734106 (529h16m33.940817141s) [upload=464207]: Uploading segments... Dec 27 16:45:34.069771537 (529h16m33.940854572s) [upload=464207]: Got next segment. Inline: false Dec 27 16:45:34.069823857 (529h16m33.940906902s) [upload=464207]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:45:34.069827965 (529h16m33.940911s) [upload=464207]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:45:34.316020208 (529h16m34.187103243s) [seg_pos={0 0}, upload=464207]: Begin upload segment... Dec 27 16:45:34.316086865 (529h16m34.18716991s) [seg_pos={0 0}, upload=464207]: Done begin upload segment. Dec 27 16:45:34.316114206 (529h16m34.187197241s) [noise=true , address=45.91.116.12:4228 , piece_id=7BB6GZGHQSIVDIV4V337YA2WF2N3VUSPQOLBVLHGHGEGSWU5YO7Q, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.317027095 (529h16m34.18811013s) [noise=true , address=145.79.227.10:4403 , piece_id=SRXUTRXFHAETBGQPAAKX5BYCRREWOKPOAFMWTJJGG3RBORN2JW6Q, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.317554920 (529h16m34.188637955s) [noise=true , address=145.79.227.11:4730 , piece_id=7UYEQXG5U3HNNYK5YGIQ6KDR2OORUAWHNYJ42MG2465QXEGMWWOQ, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.317927930 (529h16m34.189011005s) [noise=true , address=77.90.31.21:4877 , piece_id=NXSR63KJIOFACEV3ZJBIVY7EUTHLT22A2FXACY34TBRKUXQRBIKA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.318436258 (529h16m34.189519293s) [noise=true , address=145.79.227.15:4874 , piece_id=JAFPBFFD3OGMJZLFLRNKDX5SGQC52WPWYS2AGLILHDZ7E5JE5DHA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.318794620 (529h16m34.189877665s) [noise=true , address=145.79.227.51:4409 , piece_id=JH6W4GGR6XEMT4UUVEK7AQOT4MNSAJR4CFOHGBVNLCR3ZIIFE72A, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.320005085 (529h16m34.19108813s) [noise=true , address=45.91.116.14:4189 , piece_id=GQTEMSSDZBBUXFVQYIIFD7XT4NL4FKFDVQW7G3VMG4QMXQHEBBLA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.321276897 (529h16m34.192359933s) [noise=true , address=145.79.227.5:4221 , piece_id=UNP2MGLGIG7ZIRYCVJP5XV6UP5N6244CZMUPEZLSNXVBYGSJHF5Q, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.322798915 (529h16m34.19388195s) [noise=true , address=145.79.227.46:4602 , piece_id=63BRPU4J42AFCAVOXPXX4NT37662XIZXIJ4KO4AIPZWAS2U2XSGA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.324085274 (529h16m34.19516832s) [noise=true , address=77.90.31.14:4661 , piece_id=U6KYKLRNE66MELWOEAZGAKO5PU624WYV7XZYXVIHAQ2ARRLTJTGA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.325188936 (529h16m34.196271981s) [noise=true , address=145.79.227.52:4447 , piece_id=MMAOFUPHPB33H6UD5JKYP4DSXD3DQGWJLXZO3TIJE2AJHHNI3NIQ, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.325678749 (529h16m34.196761784s) [noise=true , address=145.79.227.37:4567 , piece_id=765TDBXSZCMTZC46VT2GS34CG664HQ2VJVMZMDEFLIBNF6VEBZ4A, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.327052555 (529h16m34.19813559s) [noise=true , address=45.91.116.8:4046 , piece_id=IGL3M2LM5BM5ODKJ35WFS2OVCO7IMSB6JVIE5WAYX2TIJGRFRHKQ, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.327567114 (529h16m34.198650149s) [noise=true , address=145.79.227.13:4804 , piece_id=4B2GMMMBDQ6N7P7GQZJHY2EEDMQSHMQM7B2FGVVOXYYWAKQHFDBQ, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.328792789 (529h16m34.199875824s) [noise=true , address=77.90.31.12:4590 , piece_id=2HLPEVPDS3SQI6D27AQMRAHC4PZRCAWHQO37FQHRO2MJW5DQCAQQ, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:34.329398001 (529h16m34.200481046s) [noise=true , address=45.91.116.13:4153 , piece_id=PXA6VYD4ZGPVOT4Z2OMPPMRJGY6R7T4DGIOJW4K5HK5MRHCF6YFA, seg_pos={0 0}, upload=464207]: Uploading piece... Dec 27 16:45:36.038865041 (529h16m35.909948086s) [upload=464207]: Next returned nil segment Dec 27 16:45:36.038877875 (529h16m35.90996091s) [upload=464207]: Waiting for error group managing segments... Dec 27 16:45:36.321180559 (529h16m36.192263594s) [noise=true , address=145.79.227.37:4567 , piece_id=765TDBXSZCMTZC46VT2GS34CG664HQ2VJVMZMDEFLIBNF6VEBZ4A, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.324305990 (529h16m36.195389025s) [noise=true , address=145.79.227.51:4409 , piece_id=JH6W4GGR6XEMT4UUVEK7AQOT4MNSAJR4CFOHGBVNLCR3ZIIFE72A, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.324634515 (529h16m36.19571755s) [noise=true , address=145.79.227.15:4874 , piece_id=JAFPBFFD3OGMJZLFLRNKDX5SGQC52WPWYS2AGLILHDZ7E5JE5DHA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.329053570 (529h16m36.200136605s) [noise=true , address=145.79.227.52:4447 , piece_id=MMAOFUPHPB33H6UD5JKYP4DSXD3DQGWJLXZO3TIJE2AJHHNI3NIQ, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.334638053 (529h16m36.205721098s) [noise=true , address=145.79.227.46:4602 , piece_id=63BRPU4J42AFCAVOXPXX4NT37662XIZXIJ4KO4AIPZWAS2U2XSGA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.336447619 (529h16m36.207530654s) [noise=true , address=145.79.227.11:4730 , piece_id=7UYEQXG5U3HNNYK5YGIQ6KDR2OORUAWHNYJ42MG2465QXEGMWWOQ, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.350231867 (529h16m36.221314902s) [noise=true , address=145.79.227.10:4403 , piece_id=SRXUTRXFHAETBGQPAAKX5BYCRREWOKPOAFMWTJJGG3RBORN2JW6Q, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.357434071 (529h16m36.228517116s) [noise=true , address=145.79.227.13:4804 , piece_id=4B2GMMMBDQ6N7P7GQZJHY2EEDMQSHMQM7B2FGVVOXYYWAKQHFDBQ, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.657384469 (529h16m36.528484396s) [noise=true , address=45.91.116.14:4189 , piece_id=GQTEMSSDZBBUXFVQYIIFD7XT4NL4FKFDVQW7G3VMG4QMXQHEBBLA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.673695106 (529h16m36.544778142s) [noise=true , address=45.91.116.13:4153 , piece_id=PXA6VYD4ZGPVOT4Z2OMPPMRJGY6R7T4DGIOJW4K5HK5MRHCF6YFA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.683037056 (529h16m36.554120101s) [noise=true , address=77.90.31.14:4661 , piece_id=U6KYKLRNE66MELWOEAZGAKO5PU624WYV7XZYXVIHAQ2ARRLTJTGA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:36.830363700 (529h16m36.701446735s) [noise=true , address=77.90.31.21:4877 , piece_id=NXSR63KJIOFACEV3ZJBIVY7EUTHLT22A2FXACY34TBRKUXQRBIKA, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:37.690230205 (529h16m37.56131324s) [noise=true , address=45.91.116.12:4228 , piece_id=7BB6GZGHQSIVDIV4V337YA2WF2N3VUSPQOLBVLHGHGEGSWU5YO7Q, seg_pos={0 0}, upload=464207]: Done uploading piece. err: <nil> Dec 27 16:45:37.690247017 (529h16m37.561330052s) [seg_pos={0 0}, upload=464207]: Segment reached optimal threshold of 13 pieces. Dec 27 16:45:37.690530707 (529h16m37.561613742s) [noise=true , address=77.90.31.12:4590 , piece_id=2HLPEVPDS3SQI6D27AQMRAHC4PZRCAWHQO37FQHRO2MJW5DQCAQQ, seg_pos={0 0}, upload=464207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: read tcp 188.166.225.170:53180->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 16:45:37.690538241 (529h16m37.561621276s) [noise=true , address=77.90.31.12:4590 , piece_id=2HLPEVPDS3SQI6D27AQMRAHC4PZRCAWHQO37FQHRO2MJW5DQCAQQ, seg_pos={0 0}, upload=464207]: Long tail cancellation due to optimal threshold Dec 27 16:45:37.690671705 (529h16m37.56175475s) [noise=true , address=45.91.116.8:4046 , piece_id=IGL3M2LM5BM5ODKJ35WFS2OVCO7IMSB6JVIE5WAYX2TIJGRFRHKQ, seg_pos={0 0}, upload=464207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: read tcp 188.166.225.170:57622->45.91.116.8:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 16:45:37.690679890 (529h16m37.561762935s) [noise=true , address=45.91.116.8:4046 , piece_id=IGL3M2LM5BM5ODKJ35WFS2OVCO7IMSB6JVIE5WAYX2TIJGRFRHKQ, seg_pos={0 0}, upload=464207]: Long tail cancellation due to optimal threshold Dec 27 16:45:37.690683959 (529h16m37.561766994s) [noise=true , address=145.79.227.5:4221 , piece_id=UNP2MGLGIG7ZIRYCVJP5XV6UP5N6244CZMUPEZLSNXVBYGSJHF5Q, seg_pos={0 0}, upload=464207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 27 16:45:37.690692174 (529h16m37.56177522s) [noise=true , address=145.79.227.5:4221 , piece_id=UNP2MGLGIG7ZIRYCVJP5XV6UP5N6244CZMUPEZLSNXVBYGSJHF5Q, seg_pos={0 0}, upload=464207]: Long tail cancellation due to optimal threshold Dec 27 16:45:37.690708515 (529h16m37.56179155s) [seg_pos={0 0}, upload=464207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:45:37.690779310 (529h16m37.561862345s) [upload=464207]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:45:37.690789330 (529h16m37.561872365s) [upload=464207]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:45:37.982693866 (529h16m37.853776901s) [upload=464207]: Done uploading segments... Dec 27 16:45:37.982711971 (529h16m37.853795016s) [upload=464207]: Upload finished. err: <nil> Duration: 30.003085336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......X] duration: 30.002729069s trace: https://jaeger.dev.storj.io/trace/5d1b4db99464fe84 FAIL (2025-12-27 16:45:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: TEUB54VPAZMW6HBXC33VD5A7ETZP3VW2RXAPVEAURUQEVB7YI6AQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: DSOQQHOMTO5FD765BNSAOWPYADNJK7MIFWOJ5H5DSHL4ELCIJGCA) context deadline exceeded ======LOGS====== Dec 27 16:44:58.190542770 (529h15m58.061625815s) [upload=464199]: Starting upload Dec 27 16:44:58.190627992 (529h15m58.061711037s) [upload=464199]: Done starting upload Dec 27 16:44:58.190675733 (529h15m58.061758768s) [upload=464199]: Uploading segments... Dec 27 16:44:58.190828875 (529h15m58.06191192s) [upload=464199]: Got next segment. Inline: false Dec 27 16:44:58.190860064 (529h15m58.06194311s) [upload=464199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:44:58.190864433 (529h15m58.061947478s) [upload=464199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:44:58.436419702 (529h15m58.307502757s) [seg_pos={0 0}, upload=464199]: Begin upload segment... Dec 27 16:44:58.436481099 (529h15m58.307564135s) [seg_pos={0 0}, upload=464199]: Done begin upload segment. Dec 27 16:44:58.436499143 (529h15m58.307582178s) [noise=true , address=45.91.116.8:4045 , piece_id=NAE3NPVB55FLFE2EKS3L33HNMW2WYEWK6CJYEGVBQPWVBZFMV64Q, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.436525975 (529h15m58.30760901s) [noise=true , address=145.79.227.12:4767 , piece_id=ZPW6ZCS4O7FAZPIIS5G4KVI4BDHBRSVA22BRCQRYR7XYQ3OX5I5A, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.437159881 (529h15m58.308242927s) [noise=true , address=145.79.227.3:4149 , piece_id=DKPHKV6EOS55AMML34ZGPUUHDZ7EMYO2KAA2M32OLYQAWRMGMAWQ, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.437584361 (529h15m58.308667396s) [noise=true , address=145.79.227.47:4266 , piece_id=GPVWN2RSBIKUXHDQZL3C2DDBLLO5QJEL5O6EUUERQDETSFHCU26A, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.437899380 (529h15m58.308982415s) [noise=true , address=45.91.116.10:4081 , piece_id=TEUB54VPAZMW6HBXC33VD5A7ETZP3VW2RXAPVEAURUQEVB7YI6AQ, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.438267681 (529h15m58.309350716s) [noise=true , address=145.79.227.38:4977 , piece_id=636JEEKVUYYS7ALHHII3BXE2ODRMU2PNYHK2O44L7GQKTCJGWIXQ, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.438556953 (529h15m58.309639988s) [noise=true , address=145.79.227.11:4731 , piece_id=5W3CJWLHOAZXVPYRMSMIANXRBS3LMXK3VPDTJBEDU7VKMFEQZFWQ, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.438837146 (529h15m58.309920181s) [noise=true , address=145.79.227.46:4601 , piece_id=UOPOYDG44KT3Z56GYJG4IMT6FTFZOKQZJXQRCRIR7TYHPCLYVZIQ, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.439148579 (529h15m58.310231614s) [noise=true , address=77.90.31.19:4844 , piece_id=NZTKNGHQ5QCVCZBIK3PNBVE5J3DX64VVZM2KAPCQFSF5N53IF25A, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.439479719 (529h15m58.310562764s) [noise=true , address=77.90.31.16:4735 , piece_id=ZSINMOYQWXGVWZHAFDBLJ5CUWSHBZDZ2AVAC3HEYX4FRLJS37DIA, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.440674945 (529h15m58.31175798s) [noise=true , address=145.79.227.23:4546 , piece_id=JBWCPZSTO4LMLNHLC6U7GK3Q6W2EIF63CIUNMUZJF2WIPDZ6MOJQ, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.441936598 (529h15m58.313019633s) [noise=true , address=45.91.116.12:4226 , piece_id=YRESPMONGHHZWH2H5IMYXWYJN5P3DJ65TLI3MK33AUQJ35UCPA2Q, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.443154237 (529h15m58.314237272s) [noise=true , address=145.79.227.17:4970 , piece_id=XFCXNLQ53PADYYQVZSGBVAZAHHUAC7VLOVR4B5BSVGRGYWXATFTQ, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.444399236 (529h15m58.315482272s) [noise=true , address=45.91.116.9:4261 , piece_id=DSOQQHOMTO5FD765BNSAOWPYADNJK7MIFWOJ5H5DSHL4ELCIJGCA, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.446619274 (529h15m58.317702309s) [noise=true , address=145.79.227.48:4302 , piece_id=UMMNYZGVEEFUWI2SXVX35OCURRBIYV6LKPVCBW2WOHC6XJ5EGXHQ, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:44:58.447915803 (529h15m58.318998848s) [noise=true , address=145.79.227.51:4410 , piece_id=QNOBRXIFYGB56F4YFD7PYDQMBK6ZEQTDNEE4EYNQ74UEFEVGJ4JQ, seg_pos={0 0}, upload=464199]: Uploading piece... Dec 27 16:45:00.055887729 (529h15m59.926970774s) [upload=464199]: Next returned nil segment Dec 27 16:45:00.055904360 (529h15m59.926987405s) [upload=464199]: Waiting for error group managing segments... Dec 27 16:45:00.440214884 (529h16m0.311297929s) [noise=true , address=145.79.227.17:4970 , piece_id=XFCXNLQ53PADYYQVZSGBVAZAHHUAC7VLOVR4B5BSVGRGYWXATFTQ, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.448880895 (529h16m0.31996393s) [noise=true , address=145.79.227.11:4731 , piece_id=5W3CJWLHOAZXVPYRMSMIANXRBS3LMXK3VPDTJBEDU7VKMFEQZFWQ, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.449404883 (529h16m0.320487918s) [noise=true , address=145.79.227.23:4546 , piece_id=JBWCPZSTO4LMLNHLC6U7GK3Q6W2EIF63CIUNMUZJF2WIPDZ6MOJQ, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.454106075 (529h16m0.32518911s) [noise=true , address=145.79.227.46:4601 , piece_id=UOPOYDG44KT3Z56GYJG4IMT6FTFZOKQZJXQRCRIR7TYHPCLYVZIQ, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.454500015 (529h16m0.32558306s) [noise=true , address=145.79.227.3:4149 , piece_id=DKPHKV6EOS55AMML34ZGPUUHDZ7EMYO2KAA2M32OLYQAWRMGMAWQ, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.458297856 (529h16m0.329380901s) [noise=true , address=145.79.227.48:4302 , piece_id=UMMNYZGVEEFUWI2SXVX35OCURRBIYV6LKPVCBW2WOHC6XJ5EGXHQ, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.465784483 (529h16m0.336867528s) [noise=true , address=145.79.227.47:4266 , piece_id=GPVWN2RSBIKUXHDQZL3C2DDBLLO5QJEL5O6EUUERQDETSFHCU26A, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.471256953 (529h16m0.342339988s) [noise=true , address=145.79.227.38:4977 , piece_id=636JEEKVUYYS7ALHHII3BXE2ODRMU2PNYHK2O44L7GQKTCJGWIXQ, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.475568824 (529h16m0.346651859s) [noise=true , address=145.79.227.51:4410 , piece_id=QNOBRXIFYGB56F4YFD7PYDQMBK6ZEQTDNEE4EYNQ74UEFEVGJ4JQ, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.480476118 (529h16m0.351559164s) [noise=true , address=145.79.227.12:4767 , piece_id=ZPW6ZCS4O7FAZPIIS5G4KVI4BDHBRSVA22BRCQRYR7XYQ3OX5I5A, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.617773355 (529h16m0.48885639s) [noise=true , address=77.90.31.19:4844 , piece_id=NZTKNGHQ5QCVCZBIK3PNBVE5J3DX64VVZM2KAPCQFSF5N53IF25A, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.694084926 (529h16m0.565167971s) [noise=true , address=45.91.116.9:4261 , piece_id=DSOQQHOMTO5FD765BNSAOWPYADNJK7MIFWOJ5H5DSHL4ELCIJGCA, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.793403947 (529h16m0.664486982s) [noise=true , address=45.91.116.10:4081 , piece_id=TEUB54VPAZMW6HBXC33VD5A7ETZP3VW2RXAPVEAURUQEVB7YI6AQ, seg_pos={0 0}, upload=464199]: Done uploading piece. err: <nil> Dec 27 16:45:00.793563340 (529h16m0.664646395s) [seg_pos={0 0}, upload=464199]: Segment reached optimal threshold of 13 pieces. Dec 27 16:45:00.793860195 (529h16m0.66494325s) [noise=true , address=45.91.116.8:4045 , piece_id=NAE3NPVB55FLFE2EKS3L33HNMW2WYEWK6CJYEGVBQPWVBZFMV64Q, seg_pos={0 0}, upload=464199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 16:45:00.793870705 (529h16m0.66495374s) [noise=true , address=45.91.116.8:4045 , piece_id=NAE3NPVB55FLFE2EKS3L33HNMW2WYEWK6CJYEGVBQPWVBZFMV64Q, seg_pos={0 0}, upload=464199]: Long tail cancellation due to optimal threshold Dec 27 16:45:00.793955778 (529h16m0.665038833s) [noise=true , address=45.91.116.12:4226 , piece_id=YRESPMONGHHZWH2H5IMYXWYJN5P3DJ65TLI3MK33AUQJ35UCPA2Q, seg_pos={0 0}, upload=464199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 16:45:00.793962901 (529h16m0.665045946s) [noise=true , address=45.91.116.12:4226 , piece_id=YRESPMONGHHZWH2H5IMYXWYJN5P3DJ65TLI3MK33AUQJ35UCPA2Q, seg_pos={0 0}, upload=464199]: Long tail cancellation due to optimal threshold Dec 27 16:45:00.793974813 (529h16m0.665057868s) [noise=true , address=77.90.31.16:4735 , piece_id=ZSINMOYQWXGVWZHAFDBLJ5CUWSHBZDZ2AVAC3HEYX4FRLJS37DIA, seg_pos={0 0}, upload=464199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 27 16:45:00.793983851 (529h16m0.665066916s) [noise=true , address=77.90.31.16:4735 , piece_id=ZSINMOYQWXGVWZHAFDBLJ5CUWSHBZDZ2AVAC3HEYX4FRLJS37DIA, seg_pos={0 0}, upload=464199]: Long tail cancellation due to optimal threshold Dec 27 16:45:00.794005041 (529h16m0.665088086s) [seg_pos={0 0}, upload=464199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:45:00.794095854 (529h16m0.665178909s) [upload=464199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:45:00.794103859 (529h16m0.665186904s) [upload=464199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:45:01.093404399 (529h16m0.964487464s) [upload=464199]: Done uploading segments... Dec 27 16:45:01.093429618 (529h16m0.964512663s) [upload=464199]: Upload finished. err: <nil> Duration: 30.002752245s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X.............X.....X...X.............................X..] duration: 3.550805815s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:45:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.550838146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X...X.............X.....X...X.............................X.] duration: 5.001586442s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:44:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001610588s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [XX...X.............X.....X...X.............................X] duration: 5.00129125s trace: https://jaeger.dev.storj.io/trace/67753a7736544d7b FAIL (2025-12-27 16:44:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001315226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.......] duration: 6.112377019s trace: https://jaeger.dev.storj.io/trace/4028343b04e4f8e2 FAIL (2025-12-27 16:43:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 27 16:39:10.295835987 (529h10m10.166919022s) [upload=464117]: Got next segment. Inline: false Dec 27 16:39:10.295919747 (529h10m10.167002792s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:39:10.295927441 (529h10m10.167010486s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:39:10.543273150 (529h10m10.414356185s) [seg_pos={0 0}, upload=464117]: Begin upload segment... Dec 27 16:39:10.543325019 (529h10m10.414408054s) [seg_pos={0 0}, upload=464117]: Done begin upload segment. Dec 27 16:39:10.543407706 (529h10m10.414490751s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543418046 (529h10m10.414501081s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543940150 (529h10m10.415023185s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544368226 (529h10m10.415451261s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544911670 (529h10m10.415994705s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545410931 (529h10m10.416493966s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545966639 (529h10m10.417049674s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547528382 (529h10m10.418611427s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547570874 (529h10m10.418653909s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.548887400 (529h10m10.419970445s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.550331190 (529h10m10.421414225s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.552049081 (529h10m10.423132116s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.553726535 (529h10m10.42480957s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555161618 (529h10m10.426244653s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555535359 (529h10m10.426618394s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.556454009 (529h10m10.427537044s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:12.272475108 (529h10m12.143558154s) [upload=464117]: Next returned nil segment Dec 27 16:39:12.272505146 (529h10m12.143588191s) [upload=464117]: Waiting for error group managing segments... Dec 27 16:39:12.553867307 (529h10m12.424950352s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.557002709 (529h10m12.428085744s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.562688735 (529h10m12.43377178s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.563821061 (529h10m12.434904106s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.565867748 (529h10m12.436950794s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.566613419 (529h10m12.437696464s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573191054 (529h10m12.444274099s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573577450 (529h10m12.444660495s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.586808614 (529h10m12.457891649s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.815676698 (529h10m12.686759733s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.866792832 (529h10m12.737875867s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.905056402 (529h10m12.776139437s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913484401 (529h10m12.784567436s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913511903 (529h10m12.784594948s) [seg_pos={0 0}, upload=464117]: Segment reached optimal threshold of 13 pieces. Dec 27 16:39:12.913709309 (529h10m12.784792344s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:39:12.913715861 (529h10m12.784798896s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.913973091 (529h10m12.785056126s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection Dec 27 16:39:12.913979243 (529h10m12.785062278s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914006304 (529h10m12.78508935s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection Dec 27 16:39:12.914021643 (529h10m12.785104688s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914143346 (529h10m12.785226391s) [seg_pos={0 0}, upload=464117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:39:12.914224159 (529h10m12.785307204s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:39:12.914229639 (529h10m12.785312674s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:39:13.203695743 (529h10m13.074778788s) [upload=464117]: Done uploading segments... Dec 27 16:39:13.203721502 (529h10m13.074804547s) [upload=464117]: Upload finished. err: <nil> Duration: 6.112405874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX......] duration: 8.189664931s trace: https://jaeger.dev.storj.io/trace/4028343b04e4f8e2 FAIL (2025-12-27 16:43:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 27 16:39:10.295835987 (529h10m10.166919022s) [upload=464117]: Got next segment. Inline: false Dec 27 16:39:10.295919747 (529h10m10.167002792s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:39:10.295927441 (529h10m10.167010486s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:39:10.543273150 (529h10m10.414356185s) [seg_pos={0 0}, upload=464117]: Begin upload segment... Dec 27 16:39:10.543325019 (529h10m10.414408054s) [seg_pos={0 0}, upload=464117]: Done begin upload segment. Dec 27 16:39:10.543407706 (529h10m10.414490751s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543418046 (529h10m10.414501081s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543940150 (529h10m10.415023185s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544368226 (529h10m10.415451261s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544911670 (529h10m10.415994705s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545410931 (529h10m10.416493966s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545966639 (529h10m10.417049674s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547528382 (529h10m10.418611427s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547570874 (529h10m10.418653909s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.548887400 (529h10m10.419970445s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.550331190 (529h10m10.421414225s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.552049081 (529h10m10.423132116s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.553726535 (529h10m10.42480957s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555161618 (529h10m10.426244653s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555535359 (529h10m10.426618394s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.556454009 (529h10m10.427537044s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:12.272475108 (529h10m12.143558154s) [upload=464117]: Next returned nil segment Dec 27 16:39:12.272505146 (529h10m12.143588191s) [upload=464117]: Waiting for error group managing segments... Dec 27 16:39:12.553867307 (529h10m12.424950352s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.557002709 (529h10m12.428085744s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.562688735 (529h10m12.43377178s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.563821061 (529h10m12.434904106s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.565867748 (529h10m12.436950794s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.566613419 (529h10m12.437696464s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573191054 (529h10m12.444274099s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573577450 (529h10m12.444660495s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.586808614 (529h10m12.457891649s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.815676698 (529h10m12.686759733s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.866792832 (529h10m12.737875867s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.905056402 (529h10m12.776139437s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913484401 (529h10m12.784567436s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913511903 (529h10m12.784594948s) [seg_pos={0 0}, upload=464117]: Segment reached optimal threshold of 13 pieces. Dec 27 16:39:12.913709309 (529h10m12.784792344s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:39:12.913715861 (529h10m12.784798896s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.913973091 (529h10m12.785056126s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection Dec 27 16:39:12.913979243 (529h10m12.785062278s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914006304 (529h10m12.78508935s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection Dec 27 16:39:12.914021643 (529h10m12.785104688s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914143346 (529h10m12.785226391s) [seg_pos={0 0}, upload=464117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:39:12.914224159 (529h10m12.785307204s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:39:12.914229639 (529h10m12.785312674s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:39:13.203695743 (529h10m13.074778788s) [upload=464117]: Done uploading segments... Dec 27 16:39:13.203721502 (529h10m13.074804547s) [upload=464117]: Upload finished. err: <nil> Duration: 8.189693705s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.XX...X.............X.....X...X.............................] duration: 3.848912932s trace: https://jaeger.dev.storj.io/trace/4bb0100e1f6b9a4b FAIL (2025-12-27 16:42:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.848940455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.....] duration: 6.12572665s trace: https://jaeger.dev.storj.io/trace/4028343b04e4f8e2 FAIL (2025-12-27 16:42:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 27 16:39:10.295835987 (529h10m10.166919022s) [upload=464117]: Got next segment. Inline: false Dec 27 16:39:10.295919747 (529h10m10.167002792s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:39:10.295927441 (529h10m10.167010486s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:39:10.543273150 (529h10m10.414356185s) [seg_pos={0 0}, upload=464117]: Begin upload segment... Dec 27 16:39:10.543325019 (529h10m10.414408054s) [seg_pos={0 0}, upload=464117]: Done begin upload segment. Dec 27 16:39:10.543407706 (529h10m10.414490751s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543418046 (529h10m10.414501081s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543940150 (529h10m10.415023185s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544368226 (529h10m10.415451261s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544911670 (529h10m10.415994705s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545410931 (529h10m10.416493966s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545966639 (529h10m10.417049674s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547528382 (529h10m10.418611427s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547570874 (529h10m10.418653909s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.548887400 (529h10m10.419970445s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.550331190 (529h10m10.421414225s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.552049081 (529h10m10.423132116s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.553726535 (529h10m10.42480957s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555161618 (529h10m10.426244653s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555535359 (529h10m10.426618394s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.556454009 (529h10m10.427537044s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:12.272475108 (529h10m12.143558154s) [upload=464117]: Next returned nil segment Dec 27 16:39:12.272505146 (529h10m12.143588191s) [upload=464117]: Waiting for error group managing segments... Dec 27 16:39:12.553867307 (529h10m12.424950352s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.557002709 (529h10m12.428085744s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.562688735 (529h10m12.43377178s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.563821061 (529h10m12.434904106s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.565867748 (529h10m12.436950794s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.566613419 (529h10m12.437696464s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573191054 (529h10m12.444274099s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573577450 (529h10m12.444660495s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.586808614 (529h10m12.457891649s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.815676698 (529h10m12.686759733s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.866792832 (529h10m12.737875867s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.905056402 (529h10m12.776139437s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913484401 (529h10m12.784567436s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913511903 (529h10m12.784594948s) [seg_pos={0 0}, upload=464117]: Segment reached optimal threshold of 13 pieces. Dec 27 16:39:12.913709309 (529h10m12.784792344s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:39:12.913715861 (529h10m12.784798896s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.913973091 (529h10m12.785056126s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection Dec 27 16:39:12.913979243 (529h10m12.785062278s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914006304 (529h10m12.78508935s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection Dec 27 16:39:12.914021643 (529h10m12.785104688s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914143346 (529h10m12.785226391s) [seg_pos={0 0}, upload=464117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:39:12.914224159 (529h10m12.785307204s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:39:12.914229639 (529h10m12.785312674s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:39:13.203695743 (529h10m13.074778788s) [upload=464117]: Done uploading segments... Dec 27 16:39:13.203721502 (529h10m13.074804547s) [upload=464117]: Upload finished. err: <nil> Duration: 6.125759101s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..XX...X.............X.....X...X............................] duration: 3.609986836s trace: https://jaeger.dev.storj.io/trace/4bb0100e1f6b9a4b FAIL (2025-12-27 16:42:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.610012295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X..X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX....] duration: 5.72238764s trace: https://jaeger.dev.storj.io/trace/4028343b04e4f8e2 FAIL (2025-12-27 16:42:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 27 16:39:10.295835987 (529h10m10.166919022s) [upload=464117]: Got next segment. Inline: false Dec 27 16:39:10.295919747 (529h10m10.167002792s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:39:10.295927441 (529h10m10.167010486s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:39:10.543273150 (529h10m10.414356185s) [seg_pos={0 0}, upload=464117]: Begin upload segment... Dec 27 16:39:10.543325019 (529h10m10.414408054s) [seg_pos={0 0}, upload=464117]: Done begin upload segment. Dec 27 16:39:10.543407706 (529h10m10.414490751s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543418046 (529h10m10.414501081s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543940150 (529h10m10.415023185s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544368226 (529h10m10.415451261s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544911670 (529h10m10.415994705s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545410931 (529h10m10.416493966s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545966639 (529h10m10.417049674s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547528382 (529h10m10.418611427s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547570874 (529h10m10.418653909s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.548887400 (529h10m10.419970445s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.550331190 (529h10m10.421414225s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.552049081 (529h10m10.423132116s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.553726535 (529h10m10.42480957s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555161618 (529h10m10.426244653s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555535359 (529h10m10.426618394s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.556454009 (529h10m10.427537044s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:12.272475108 (529h10m12.143558154s) [upload=464117]: Next returned nil segment Dec 27 16:39:12.272505146 (529h10m12.143588191s) [upload=464117]: Waiting for error group managing segments... Dec 27 16:39:12.553867307 (529h10m12.424950352s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.557002709 (529h10m12.428085744s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.562688735 (529h10m12.43377178s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.563821061 (529h10m12.434904106s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.565867748 (529h10m12.436950794s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.566613419 (529h10m12.437696464s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573191054 (529h10m12.444274099s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573577450 (529h10m12.444660495s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.586808614 (529h10m12.457891649s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.815676698 (529h10m12.686759733s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.866792832 (529h10m12.737875867s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.905056402 (529h10m12.776139437s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913484401 (529h10m12.784567436s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913511903 (529h10m12.784594948s) [seg_pos={0 0}, upload=464117]: Segment reached optimal threshold of 13 pieces. Dec 27 16:39:12.913709309 (529h10m12.784792344s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:39:12.913715861 (529h10m12.784798896s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.913973091 (529h10m12.785056126s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection Dec 27 16:39:12.913979243 (529h10m12.785062278s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914006304 (529h10m12.78508935s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection Dec 27 16:39:12.914021643 (529h10m12.785104688s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914143346 (529h10m12.785226391s) [seg_pos={0 0}, upload=464117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:39:12.914224159 (529h10m12.785307204s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:39:12.914229639 (529h10m12.785312674s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:39:13.203695743 (529h10m13.074778788s) [upload=464117]: Done uploading segments... Dec 27 16:39:13.203721502 (529h10m13.074804547s) [upload=464117]: Upload finished. err: <nil> Duration: 5.722449758s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX..X..X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX...] duration: 5.71451573s trace: https://jaeger.dev.storj.io/trace/4028343b04e4f8e2 FAIL (2025-12-27 16:41:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 27 16:39:10.295835987 (529h10m10.166919022s) [upload=464117]: Got next segment. Inline: false Dec 27 16:39:10.295919747 (529h10m10.167002792s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:39:10.295927441 (529h10m10.167010486s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:39:10.543273150 (529h10m10.414356185s) [seg_pos={0 0}, upload=464117]: Begin upload segment... Dec 27 16:39:10.543325019 (529h10m10.414408054s) [seg_pos={0 0}, upload=464117]: Done begin upload segment. Dec 27 16:39:10.543407706 (529h10m10.414490751s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543418046 (529h10m10.414501081s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543940150 (529h10m10.415023185s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544368226 (529h10m10.415451261s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544911670 (529h10m10.415994705s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545410931 (529h10m10.416493966s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545966639 (529h10m10.417049674s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547528382 (529h10m10.418611427s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547570874 (529h10m10.418653909s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.548887400 (529h10m10.419970445s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.550331190 (529h10m10.421414225s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.552049081 (529h10m10.423132116s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.553726535 (529h10m10.42480957s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555161618 (529h10m10.426244653s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555535359 (529h10m10.426618394s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.556454009 (529h10m10.427537044s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:12.272475108 (529h10m12.143558154s) [upload=464117]: Next returned nil segment Dec 27 16:39:12.272505146 (529h10m12.143588191s) [upload=464117]: Waiting for error group managing segments... Dec 27 16:39:12.553867307 (529h10m12.424950352s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.557002709 (529h10m12.428085744s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.562688735 (529h10m12.43377178s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.563821061 (529h10m12.434904106s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.565867748 (529h10m12.436950794s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.566613419 (529h10m12.437696464s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573191054 (529h10m12.444274099s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573577450 (529h10m12.444660495s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.586808614 (529h10m12.457891649s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.815676698 (529h10m12.686759733s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.866792832 (529h10m12.737875867s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.905056402 (529h10m12.776139437s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913484401 (529h10m12.784567436s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913511903 (529h10m12.784594948s) [seg_pos={0 0}, upload=464117]: Segment reached optimal threshold of 13 pieces. Dec 27 16:39:12.913709309 (529h10m12.784792344s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:39:12.913715861 (529h10m12.784798896s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.913973091 (529h10m12.785056126s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection Dec 27 16:39:12.913979243 (529h10m12.785062278s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914006304 (529h10m12.78508935s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection Dec 27 16:39:12.914021643 (529h10m12.785104688s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914143346 (529h10m12.785226391s) [seg_pos={0 0}, upload=464117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:39:12.914224159 (529h10m12.785307204s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:39:12.914229639 (529h10m12.785312674s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:39:13.203695743 (529h10m13.074778788s) [upload=464117]: Done uploading segments... Dec 27 16:39:13.203721502 (529h10m13.074804547s) [upload=464117]: Upload finished. err: <nil> Duration: 5.71454176s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...XX...X.............X.....X...X...........................] duration: 3.683124563s trace: https://jaeger.dev.storj.io/trace/4bb0100e1f6b9a4b FAIL (2025-12-27 16:41:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.683211839s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....XX...X.............X.....X...X..........................] duration: 4.160245043s trace: https://jaeger.dev.storj.io/trace/4bb0100e1f6b9a4b FAIL (2025-12-27 16:40:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.160272786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX..X..X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX..] duration: 5.858891047s trace: https://jaeger.dev.storj.io/trace/4028343b04e4f8e2 FAIL (2025-12-27 16:40:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 27 16:39:10.295835987 (529h10m10.166919022s) [upload=464117]: Got next segment. Inline: false Dec 27 16:39:10.295919747 (529h10m10.167002792s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:39:10.295927441 (529h10m10.167010486s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:39:10.543273150 (529h10m10.414356185s) [seg_pos={0 0}, upload=464117]: Begin upload segment... Dec 27 16:39:10.543325019 (529h10m10.414408054s) [seg_pos={0 0}, upload=464117]: Done begin upload segment. Dec 27 16:39:10.543407706 (529h10m10.414490751s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543418046 (529h10m10.414501081s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543940150 (529h10m10.415023185s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544368226 (529h10m10.415451261s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544911670 (529h10m10.415994705s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545410931 (529h10m10.416493966s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545966639 (529h10m10.417049674s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547528382 (529h10m10.418611427s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547570874 (529h10m10.418653909s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.548887400 (529h10m10.419970445s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.550331190 (529h10m10.421414225s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.552049081 (529h10m10.423132116s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.553726535 (529h10m10.42480957s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555161618 (529h10m10.426244653s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555535359 (529h10m10.426618394s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.556454009 (529h10m10.427537044s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:12.272475108 (529h10m12.143558154s) [upload=464117]: Next returned nil segment Dec 27 16:39:12.272505146 (529h10m12.143588191s) [upload=464117]: Waiting for error group managing segments... Dec 27 16:39:12.553867307 (529h10m12.424950352s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.557002709 (529h10m12.428085744s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.562688735 (529h10m12.43377178s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.563821061 (529h10m12.434904106s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.565867748 (529h10m12.436950794s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.566613419 (529h10m12.437696464s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573191054 (529h10m12.444274099s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573577450 (529h10m12.444660495s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.586808614 (529h10m12.457891649s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.815676698 (529h10m12.686759733s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.866792832 (529h10m12.737875867s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.905056402 (529h10m12.776139437s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913484401 (529h10m12.784567436s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913511903 (529h10m12.784594948s) [seg_pos={0 0}, upload=464117]: Segment reached optimal threshold of 13 pieces. Dec 27 16:39:12.913709309 (529h10m12.784792344s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:39:12.913715861 (529h10m12.784798896s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.913973091 (529h10m12.785056126s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection Dec 27 16:39:12.913979243 (529h10m12.785062278s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914006304 (529h10m12.78508935s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection Dec 27 16:39:12.914021643 (529h10m12.785104688s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914143346 (529h10m12.785226391s) [seg_pos={0 0}, upload=464117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:39:12.914224159 (529h10m12.785307204s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:39:12.914229639 (529h10m12.785312674s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:39:13.203695743 (529h10m13.074778788s) [upload=464117]: Done uploading segments... Dec 27 16:39:13.203721502 (529h10m13.074804547s) [upload=464117]: Upload finished. err: <nil> Duration: 5.858922356s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....XX...X.............X.....X...X.........................] duration: 3.914586256s trace: https://jaeger.dev.storj.io/trace/4bb0100e1f6b9a4b FAIL (2025-12-27 16:40:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.914612816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.XX..X..X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX.] duration: 8.740335652s trace: https://jaeger.dev.storj.io/trace/4028343b04e4f8e2 FAIL (2025-12-27 16:40:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 27 16:39:10.295835987 (529h10m10.166919022s) [upload=464117]: Got next segment. Inline: false Dec 27 16:39:10.295919747 (529h10m10.167002792s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:39:10.295927441 (529h10m10.167010486s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:39:10.543273150 (529h10m10.414356185s) [seg_pos={0 0}, upload=464117]: Begin upload segment... Dec 27 16:39:10.543325019 (529h10m10.414408054s) [seg_pos={0 0}, upload=464117]: Done begin upload segment. Dec 27 16:39:10.543407706 (529h10m10.414490751s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543418046 (529h10m10.414501081s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543940150 (529h10m10.415023185s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544368226 (529h10m10.415451261s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544911670 (529h10m10.415994705s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545410931 (529h10m10.416493966s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545966639 (529h10m10.417049674s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547528382 (529h10m10.418611427s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547570874 (529h10m10.418653909s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.548887400 (529h10m10.419970445s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.550331190 (529h10m10.421414225s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.552049081 (529h10m10.423132116s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.553726535 (529h10m10.42480957s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555161618 (529h10m10.426244653s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555535359 (529h10m10.426618394s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.556454009 (529h10m10.427537044s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:12.272475108 (529h10m12.143558154s) [upload=464117]: Next returned nil segment Dec 27 16:39:12.272505146 (529h10m12.143588191s) [upload=464117]: Waiting for error group managing segments... Dec 27 16:39:12.553867307 (529h10m12.424950352s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.557002709 (529h10m12.428085744s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.562688735 (529h10m12.43377178s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.563821061 (529h10m12.434904106s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.565867748 (529h10m12.436950794s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.566613419 (529h10m12.437696464s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573191054 (529h10m12.444274099s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573577450 (529h10m12.444660495s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.586808614 (529h10m12.457891649s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.815676698 (529h10m12.686759733s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.866792832 (529h10m12.737875867s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.905056402 (529h10m12.776139437s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913484401 (529h10m12.784567436s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913511903 (529h10m12.784594948s) [seg_pos={0 0}, upload=464117]: Segment reached optimal threshold of 13 pieces. Dec 27 16:39:12.913709309 (529h10m12.784792344s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:39:12.913715861 (529h10m12.784798896s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.913973091 (529h10m12.785056126s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection Dec 27 16:39:12.913979243 (529h10m12.785062278s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914006304 (529h10m12.78508935s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection Dec 27 16:39:12.914021643 (529h10m12.785104688s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914143346 (529h10m12.785226391s) [seg_pos={0 0}, upload=464117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:39:12.914224159 (529h10m12.785307204s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:39:12.914229639 (529h10m12.785312674s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:39:13.203695743 (529h10m13.074778788s) [upload=464117]: Done uploading segments... Dec 27 16:39:13.203721502 (529h10m13.074804547s) [upload=464117]: Upload finished. err: <nil> Duration: 8.740366431s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......XX...X.............X.....X...X........................] duration: 3.532271233s trace: https://jaeger.dev.storj.io/trace/4bb0100e1f6b9a4b FAIL (2025-12-27 16:39:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532294857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 40.829075427s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:39:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 40.829109082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X.XX..X..X....XX.X....X..X.XXX..X.XX.X..X..XXXX.XX.......XX] duration: 30.003339973s trace: https://jaeger.dev.storj.io/trace/4028343b04e4f8e2 FAIL (2025-12-27 16:39:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 27 16:39:10.295835987 (529h10m10.166919022s) [upload=464117]: Got next segment. Inline: false Dec 27 16:39:10.295919747 (529h10m10.167002792s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:39:10.295927441 (529h10m10.167010486s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:39:10.543273150 (529h10m10.414356185s) [seg_pos={0 0}, upload=464117]: Begin upload segment... Dec 27 16:39:10.543325019 (529h10m10.414408054s) [seg_pos={0 0}, upload=464117]: Done begin upload segment. Dec 27 16:39:10.543407706 (529h10m10.414490751s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543418046 (529h10m10.414501081s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.543940150 (529h10m10.415023185s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544368226 (529h10m10.415451261s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.544911670 (529h10m10.415994705s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545410931 (529h10m10.416493966s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.545966639 (529h10m10.417049674s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547528382 (529h10m10.418611427s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.547570874 (529h10m10.418653909s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.548887400 (529h10m10.419970445s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.550331190 (529h10m10.421414225s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.552049081 (529h10m10.423132116s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.553726535 (529h10m10.42480957s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555161618 (529h10m10.426244653s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.555535359 (529h10m10.426618394s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:10.556454009 (529h10m10.427537044s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Uploading piece... Dec 27 16:39:12.272475108 (529h10m12.143558154s) [upload=464117]: Next returned nil segment Dec 27 16:39:12.272505146 (529h10m12.143588191s) [upload=464117]: Waiting for error group managing segments... Dec 27 16:39:12.553867307 (529h10m12.424950352s) [noise=true , address=145.79.227.50:4374 , piece_id=H73UMQVHB5HXZACULHZXNEUJAY3C54I7Q3KDXLPFFZMUCSAR7GIQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.557002709 (529h10m12.428085744s) [noise=true , address=145.79.227.4:4185 , piece_id=JWQ37OJMPRFE3U45K2QUSVLTH55QAAAIOH24ZHTIUEDOXLP6SOCA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.562688735 (529h10m12.43377178s) [noise=true , address=145.79.227.41:4087 , piece_id=BU5OIC4C35NTYT26BKP5D6LHGSI6QRTBHYSBEOIIRXMQWQQGBVFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.563821061 (529h10m12.434904106s) [noise=true , address=145.79.227.8:4329 , piece_id=DWQPXZMJP6XBM7OFIZLUXIYLW253EVWLC554JDW5IIIPW3CP6MKQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.565867748 (529h10m12.436950794s) [noise=true , address=145.79.227.2:4113 , piece_id=QSQ3OAOLGAMUWKNNL4ETDXZWPHQ7KOTJ2IEY4JNXO5ZWDBOQPP5Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.566613419 (529h10m12.437696464s) [noise=true , address=145.79.227.19:4078 , piece_id=CBZRRATR3RXVMSONSUN4DVON3PSI7MU36A4PWCJADO2ATJMVQMTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573191054 (529h10m12.444274099s) [noise=true , address=145.79.227.47:4265 , piece_id=OLLPGCVZQFSPDFQCKFX2WIQ5AJC7Z6NNJPX3AQDDBVZE2YOKHTEA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.573577450 (529h10m12.444660495s) [noise=true , address=145.79.227.15:4876 , piece_id=YYCSXSURMJQERUP3WPRLFIVWHMKVXVBZZZA7CSCAFP67BAW2RQWQ, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.586808614 (529h10m12.457891649s) [noise=true , address=145.79.227.39:4013 , piece_id=NIQLTBCNXEZU4AGHC6MNG6N7MUL543WIB7E7DQLL4S5OG6O5QX2Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.815676698 (529h10m12.686759733s) [noise=true , address=45.91.116.5:4300 , piece_id=MJ6XRA2CBP74BZ63ZOVUT7OHFREGHRVUA2MLXLW6RI5IDCETSZTA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.866792832 (529h10m12.737875867s) [noise=true , address=45.91.116.6:4974 , piece_id=5XG37KXYWCQ3YQ4GO234M7APNKNVBCDY3SFCSDGHSYZUVLBZDU6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.905056402 (529h10m12.776139437s) [noise=true , address=45.91.116.8:4046 , piece_id=PTUZRI3T4BSMWA4BRELPKFH4ZOD7HCRVZ62ML5VV2SSR277LSALA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913484401 (529h10m12.784567436s) [noise=true , address=45.91.116.7:4011 , piece_id=FE7354DJII2JZZ5OJT76PQ464ASCNQ7HVBKIL4KA3TIXYHE77L7A, seg_pos={0 0}, upload=464117]: Done uploading piece. err: <nil> Dec 27 16:39:12.913511903 (529h10m12.784594948s) [seg_pos={0 0}, upload=464117]: Segment reached optimal threshold of 13 pieces. Dec 27 16:39:12.913709309 (529h10m12.784792344s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:39:12.913715861 (529h10m12.784798896s) [noise=true , address=45.91.116.13:4156 , piece_id=UMZ4IOL2QM5KE5Q7MX6L3RBHB5RIRVTS6PWAIJJ6QWRSZS6ITEFA, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.913973091 (529h10m12.785056126s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48604->77.90.31.13:4627: use of closed network connection Dec 27 16:39:12.913979243 (529h10m12.785062278s) [noise=true , address=77.90.31.13:4627 , piece_id=62EQZPN7KXWOQ35JJ77SVDWVHPPDCVWVJJFYVI6L67AZROUWEP6Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914006304 (529h10m12.78508935s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60110->77.90.31.19:4842: use of closed network connection Dec 27 16:39:12.914021643 (529h10m12.785104688s) [noise=true , address=77.90.31.19:4842 , piece_id=CFM2NMTBN6ES6EO344JQSXLO77TJVD7WL5I62TA4DTKM6DXMOY7Q, seg_pos={0 0}, upload=464117]: Long tail cancellation due to optimal threshold Dec 27 16:39:12.914143346 (529h10m12.785226391s) [seg_pos={0 0}, upload=464117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:39:12.914224159 (529h10m12.785307204s) [upload=464117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:39:12.914229639 (529h10m12.785312674s) [upload=464117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:39:13.203695743 (529h10m13.074778788s) [upload=464117]: Done uploading segments... Dec 27 16:39:13.203721502 (529h10m13.074804547s) [upload=464117]: Upload finished. err: <nil> Duration: 30.003350473s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X......XX...X.............X.....X...X.......................] duration: 3.493893584s trace: https://jaeger.dev.storj.io/trace/4bb0100e1f6b9a4b FAIL (2025-12-27 16:39:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.493916258s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)