Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 20.97303007s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:25:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 20.973058805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.X.XX.X.XXX] duration: 30.003181255s trace: https://jaeger.dev.storj.io/trace/453ed28c614f1d2c FAIL (2025-12-28 00:25:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0:24:36.657470819 (536h55m36.528553854s) [upload=470883]: Starting upload Dec 28 00:24:36.657584656 (536h55m36.528667701s) [upload=470883]: Done starting upload Dec 28 00:24:36.657648357 (536h55m36.528731402s) [upload=470883]: Uploading segments... Dec 28 00:24:36.657961093 (536h55m36.529044138s) [upload=470883]: Got next segment. Inline: false Dec 28 00:24:36.658018523 (536h55m36.529101568s) [upload=470883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:24:36.658048068 (536h55m36.529131103s) [upload=470883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:24:36.902046602 (536h55m36.773129637s) [seg_pos={0 0}, upload=470883]: Begin upload segment... Dec 28 00:24:36.902096467 (536h55m36.773179512s) [seg_pos={0 0}, upload=470883]: Done begin upload segment. Dec 28 00:24:36.902154558 (536h55m36.773237593s) [noise=true , address=145.79.227.4:4186 , piece_id=RRP5Y4STUE2EFM2HI6XCQTJ4KURYB5VWDQFUFZSNL6FOD2J4TWBA, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.902272011 (536h55m36.773355046s) [noise=true , address=145.79.227.15:4874 , piece_id=JLQIH3HXAQTVREKMYXCBYN4BGBI63GGX3HLKRMDQHMI2UEV3A4BQ, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.902993335 (536h55m36.77407638s) [noise=true , address=45.91.116.14:4191 , piece_id=TJXIADGGQBHZUTUKPIDDNVOO7J6LG3VEYBZYOA25BRMVGVXTQNBQ, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.903480723 (536h55m36.774563768s) [noise=true , address=45.91.116.13:4156 , piece_id=H6UYNQV24MNFCOSKUXVLE2PNNU4YYEFYCAZU33HMICSN6Y6UFLQA, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.904005211 (536h55m36.775088246s) [noise=true , address=145.79.227.19:4079 , piece_id=GW5NFBKWVNBEGQCU5SDYE3Z722D75DFA5COFMNDAIH7BRVO645PA, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.904483092 (536h55m36.775566137s) [noise=true , address=45.91.116.8:4045 , piece_id=GBZRNTYD3KWHLL37O467ODJYWXTAS34T662R67KTWQG7RR62LY6Q, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.904958197 (536h55m36.776041242s) [noise=true , address=45.91.116.11:4117 , piece_id=6IOY66YWLIAXGYSXYB5HSKCPRSUNZPCDFZRMEX7L3TQWSYC5OAAA, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.905468708 (536h55m36.776551743s) [noise=true , address=45.91.116.6:4976 , piece_id=3WPQKHZUYRZQ7WVLRYDHPVACS5VDKVKK63X27AH7D5MTJPPRHYCQ, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.907094404 (536h55m36.778177439s) [noise=true , address=145.79.227.47:4265 , piece_id=VB3U27HDK2D4K2EBEIT36QULBKEU55NY6LQJXFDF5XON4FL3LJUA, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.907697532 (536h55m36.778780577s) [noise=true , address=145.79.227.37:4566 , piece_id=L2MKN5TFBVQY6GEBXIXAJBDR6XOIWCK4D6RRHZYA5553QUGT5MTA, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.908567338 (536h55m36.779650383s) [noise=true , address=145.79.227.50:4375 , piece_id=C7KJOCTHUTTXJB5QFCYW2ON3LPESIJ3U5J4JJID72N4UIN2ICBWA, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.910190168 (536h55m36.781273203s) [noise=true , address=145.79.227.9:4365 , piece_id=HIMG2QVT3BZ3ZLH7VRNMALEHWPQ7RRVN2QFCKNUHM34YVFPXZEBA, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.912446364 (536h55m36.783529399s) [noise=true , address=145.79.227.51:4410 , piece_id=7BG3O5TCQLVTMED6CXVCRPEIUBBMQQASFHIWPCPXHYRFCCIYJ55Q, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.914219030 (536h55m36.785302075s) [noise=true , address=145.79.227.52:4446 , piece_id=PCFYZ7FQYVCYRZPL4QUFQ4ZBRVQY5OIALFE2EG4QKRFYVCUFG76Q, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.918739097 (536h55m36.789822132s) [noise=true , address=145.79.227.16:4910 , piece_id=PE2UHVMR2OG6H4XVAEXJTROXKDB425NXJUEX3Z4DRJFY4VJQRWJQ, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:36.920197755 (536h55m36.7912808s) [noise=true , address=145.79.227.24:4581 , piece_id=IG42NANXOBHXXDZHOR6T2XQ2MEYXSDOZXT4EA5N4JFIMOXDWJZZA, seg_pos={0 0}, upload=470883]: Uploading piece... Dec 28 00:24:38.630327700 (536h55m38.501410735s) [upload=470883]: Next returned nil segment Dec 28 00:24:38.630417962 (536h55m38.501501017s) [upload=470883]: Waiting for error group managing segments... Dec 28 00:24:38.917903962 (536h55m38.788987008s) [noise=true , address=145.79.227.15:4874 , piece_id=JLQIH3HXAQTVREKMYXCBYN4BGBI63GGX3HLKRMDQHMI2UEV3A4BQ, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:38.926566868 (536h55m38.797649903s) [noise=true , address=145.79.227.37:4566 , piece_id=L2MKN5TFBVQY6GEBXIXAJBDR6XOIWCK4D6RRHZYA5553QUGT5MTA, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:38.933948603 (536h55m38.805038591s) [noise=true , address=145.79.227.9:4365 , piece_id=HIMG2QVT3BZ3ZLH7VRNMALEHWPQ7RRVN2QFCKNUHM34YVFPXZEBA, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:38.934016873 (536h55m38.805099908s) [noise=true , address=145.79.227.50:4375 , piece_id=C7KJOCTHUTTXJB5QFCYW2ON3LPESIJ3U5J4JJID72N4UIN2ICBWA, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:38.939135530 (536h55m38.810218575s) [noise=true , address=145.79.227.4:4186 , piece_id=RRP5Y4STUE2EFM2HI6XCQTJ4KURYB5VWDQFUFZSNL6FOD2J4TWBA, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:38.942026415 (536h55m38.81310946s) [noise=true , address=145.79.227.24:4581 , piece_id=IG42NANXOBHXXDZHOR6T2XQ2MEYXSDOZXT4EA5N4JFIMOXDWJZZA, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:38.952220156 (536h55m38.823303191s) [noise=true , address=145.79.227.52:4446 , piece_id=PCFYZ7FQYVCYRZPL4QUFQ4ZBRVQY5OIALFE2EG4QKRFYVCUFG76Q, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:39.119522697 (536h55m38.990605733s) [noise=true , address=145.79.227.51:4410 , piece_id=7BG3O5TCQLVTMED6CXVCRPEIUBBMQQASFHIWPCPXHYRFCCIYJ55Q, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:39.127800619 (536h55m38.998883654s) [noise=true , address=45.91.116.14:4191 , piece_id=TJXIADGGQBHZUTUKPIDDNVOO7J6LG3VEYBZYOA25BRMVGVXTQNBQ, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:39.172701038 (536h55m39.043784083s) [noise=true , address=45.91.116.13:4156 , piece_id=H6UYNQV24MNFCOSKUXVLE2PNNU4YYEFYCAZU33HMICSN6Y6UFLQA, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:39.365657157 (536h55m39.236740202s) [noise=true , address=145.79.227.47:4265 , piece_id=VB3U27HDK2D4K2EBEIT36QULBKEU55NY6LQJXFDF5XON4FL3LJUA, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:39.879041164 (536h55m39.750124199s) [noise=true , address=45.91.116.11:4117 , piece_id=6IOY66YWLIAXGYSXYB5HSKCPRSUNZPCDFZRMEX7L3TQWSYC5OAAA, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:40.077638917 (536h55m39.948721952s) [noise=true , address=45.91.116.6:4976 , piece_id=3WPQKHZUYRZQ7WVLRYDHPVACS5VDKVKK63X27AH7D5MTJPPRHYCQ, seg_pos={0 0}, upload=470883]: Done uploading piece. err: <nil> Dec 28 00:24:40.077661440 (536h55m39.948744475s) [seg_pos={0 0}, upload=470883]: Segment reached optimal threshold of 13 pieces. Dec 28 00:24:40.077892490 (536h55m39.948975525s) [noise=true , address=45.91.116.8:4045 , piece_id=GBZRNTYD3KWHLL37O467ODJYWXTAS34T662R67KTWQG7RR62LY6Q, seg_pos={0 0}, upload=470883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 00:24:40.077900195 (536h55m39.94898323s) [noise=true , address=45.91.116.8:4045 , piece_id=GBZRNTYD3KWHLL37O467ODJYWXTAS34T662R67KTWQG7RR62LY6Q, seg_pos={0 0}, upload=470883]: Long tail cancellation due to optimal threshold Dec 28 00:24:40.077993351 (536h55m39.949076387s) [noise=true , address=145.79.227.19:4079 , piece_id=GW5NFBKWVNBEGQCU5SDYE3Z722D75DFA5COFMNDAIH7BRVO645PA, seg_pos={0 0}, upload=470883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 00:24:40.077997971 (536h55m39.949081006s) [noise=true , address=145.79.227.19:4079 , piece_id=GW5NFBKWVNBEGQCU5SDYE3Z722D75DFA5COFMNDAIH7BRVO645PA, seg_pos={0 0}, upload=470883]: Long tail cancellation due to optimal threshold Dec 28 00:24:40.078088162 (536h55m39.949171217s) [noise=true , address=145.79.227.16:4910 , piece_id=PE2UHVMR2OG6H4XVAEXJTROXKDB425NXJUEX3Z4DRJFY4VJQRWJQ, seg_pos={0 0}, upload=470883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:48924->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:48924->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48924->145.79.227.16:4910: use of closed network connection Dec 28 00:24:40.078106738 (536h55m39.949189773s) [noise=true , address=145.79.227.16:4910 , piece_id=PE2UHVMR2OG6H4XVAEXJTROXKDB425NXJUEX3Z4DRJFY4VJQRWJQ, seg_pos={0 0}, upload=470883]: Long tail cancellation due to optimal threshold Dec 28 00:24:40.078147836 (536h55m39.949230871s) [seg_pos={0 0}, upload=470883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:24:40.078265309 (536h55m39.949348344s) [upload=470883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:24:40.078271521 (536h55m39.949354556s) [upload=470883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:24:40.372440438 (536h55m40.243523473s) [upload=470883]: Done uploading segments... Dec 28 00:24:40.372458572 (536h55m40.243541607s) [upload=470883]: Upload finished. err: <nil> Duration: 30.003192926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.X.XX.X.XX] duration: 30.003447808s trace: https://jaeger.dev.storj.io/trace/b1c7ace4be100f FAIL (2025-12-28 00:24:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: GSSCGNP5VDSO7UFLSXQJPZR6MXENZSBLNURH5VDAMG5AL6BKCKJA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: YGIUUZ6MIM6AQRGQTBXTIEHUIGXNFFWL456MMW36L567REVI5GZA) context deadline exceeded ======LOGS====== Dec 28 00:24:06.432834682 (536h55m6.303917717s) [upload=470877]: Starting upload Dec 28 00:24:06.432918461 (536h55m6.304001506s) [upload=470877]: Done starting upload Dec 28 00:24:06.433053017 (536h55m6.304136052s) [upload=470877]: Uploading segments... Dec 28 00:24:06.433176182 (536h55m6.304259227s) [upload=470877]: Got next segment. Inline: false Dec 28 00:24:06.433224714 (536h55m6.304307759s) [upload=470877]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:24:06.433231537 (536h55m6.304314582s) [upload=470877]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:24:06.678929384 (536h55m6.550012429s) [seg_pos={0 0}, upload=470877]: Begin upload segment... Dec 28 00:24:06.678966184 (536h55m6.550049219s) [seg_pos={0 0}, upload=470877]: Done begin upload segment. Dec 28 00:24:06.678987525 (536h55m6.55007057s) [noise=true , address=45.91.116.13:4156 , piece_id=UK6PAHAPXAEJ3TPZ6NMEMSQGQAGL6UHKDBY6DLVZRW6FSFZJU4ZA, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.684298628 (536h55m6.555381673s) [noise=true , address=145.79.227.49:4337 , piece_id=WFDMFSR4PBEPBWA2HRTSXUEM3DNKQ3NIWKIHODXRKHQDLORSGB2A, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.686024204 (536h55m6.557107249s) [noise=true , address=145.79.227.48:4302 , piece_id=AX4KKFVL42L54PN2MZFUZU5NC45HMI3ANR7NC7U2ZYYRVU5NZQ5A, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.687984797 (536h55m6.559067832s) [noise=true , address=45.91.116.12:4226 , piece_id=YGIUUZ6MIM6AQRGQTBXTIEHUIGXNFFWL456MMW36L567REVI5GZA, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.691462990 (536h55m6.562546035s) [noise=true , address=77.90.31.16:4736 , piece_id=VKGQU254DOXYQZNTM6I2P6NGK3MC4YV3PALZJSCLRPSRLVKJII4A, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.692211496 (536h55m6.563294541s) [noise=true , address=45.91.116.14:4190 , piece_id=S2XD7GJQK2YKBD2W7ESEWK6ABRQTUZJSH5FXFIRV3QCDI3LHA6BA, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.695599447 (536h55m6.566682482s) [noise=true , address=145.79.227.5:4221 , piece_id=YILTMQEE7TRXEPZM74LGIOTRSGA44DGPPDHHVCSCIJ3UCWGIT3UA, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.696875798 (536h55m6.567958843s) [noise=true , address=145.79.227.33:4871 , piece_id=Y2N5VCAWGQDFR6OTM6FYIXNQKPIWB2FU3CZMA63TVNAX7GQO3NJA, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.698907065 (536h55m6.56999011s) [noise=true , address=45.91.116.4:4937 , piece_id=2YSWMFDVAPEL4RZVKLA2S77VL2RRWYSNC57VCCLYGU55ICDJWZBQ, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.700183476 (536h55m6.571266511s) [noise=true , address=145.79.227.52:4447 , piece_id=QLHXCGRPAB5D6YXTT2UUXJUTBIUNCRPJTI4FGP5VJKLA5XC6JELQ, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.705540998 (536h55m6.576624033s) [noise=true , address=145.79.227.41:4087 , piece_id=WPPY7A2FLQHRP2Q37WOGURU72R7TA36NWC6WHDJZZOQM22RH3VZQ, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.707365532 (536h55m6.578448577s) [noise=true , address=77.90.31.13:4629 , piece_id=SOTAFTXY3WGKKFB5WY4KQX7TH3NQ56X2OOHLDZ4I33ZP7SAVID4A, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.708996978 (536h55m6.580080023s) [noise=true , address=77.90.31.18:4805 , piece_id=GSSCGNP5VDSO7UFLSXQJPZR6MXENZSBLNURH5VDAMG5AL6BKCKJA, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.710204197 (536h55m6.581287242s) [noise=true , address=145.79.227.11:4729 , piece_id=EKWMJ2M3ODEWQ72JD2QTHQCKIWJSLEKQBCR3ZKUOIRQIPILQKXAQ, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.711525724 (536h55m6.582608769s) [noise=true , address=45.91.116.10:4081 , piece_id=BIR7PSNHRU2UUSNEKMHEB52HX5EJATQ2OOJ6WRGLABRB5ZPSQI5Q, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:06.712750095 (536h55m6.58383314s) [noise=true , address=145.79.227.36:4519 , piece_id=XOW667AVPIWLK3QMPNQFRYWKTT3NCBKOFR6E5EDR7Y6275ZSGZ6Q, seg_pos={0 0}, upload=470877]: Uploading piece... Dec 28 00:24:08.424038117 (536h55m8.295121162s) [upload=470877]: Next returned nil segment Dec 28 00:24:08.424055880 (536h55m8.295138916s) [upload=470877]: Waiting for error group managing segments... Dec 28 00:24:08.727625226 (536h55m8.598708271s) [noise=true , address=145.79.227.48:4302 , piece_id=AX4KKFVL42L54PN2MZFUZU5NC45HMI3ANR7NC7U2ZYYRVU5NZQ5A, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:08.728323586 (536h55m8.599406631s) [noise=true , address=145.79.227.41:4087 , piece_id=WPPY7A2FLQHRP2Q37WOGURU72R7TA36NWC6WHDJZZOQM22RH3VZQ, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:08.739656446 (536h55m8.610739491s) [noise=true , address=145.79.227.5:4221 , piece_id=YILTMQEE7TRXEPZM74LGIOTRSGA44DGPPDHHVCSCIJ3UCWGIT3UA, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:08.743970420 (536h55m8.615053455s) [noise=true , address=145.79.227.11:4729 , piece_id=EKWMJ2M3ODEWQ72JD2QTHQCKIWJSLEKQBCR3ZKUOIRQIPILQKXAQ, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:08.747468071 (536h55m8.618551106s) [noise=true , address=145.79.227.36:4519 , piece_id=XOW667AVPIWLK3QMPNQFRYWKTT3NCBKOFR6E5EDR7Y6275ZSGZ6Q, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:08.762179944 (536h55m8.633262989s) [noise=true , address=145.79.227.52:4447 , piece_id=QLHXCGRPAB5D6YXTT2UUXJUTBIUNCRPJTI4FGP5VJKLA5XC6JELQ, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:08.936625257 (536h55m8.807708302s) [noise=true , address=77.90.31.18:4805 , piece_id=GSSCGNP5VDSO7UFLSXQJPZR6MXENZSBLNURH5VDAMG5AL6BKCKJA, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:08.946325106 (536h55m8.817408151s) [noise=true , address=77.90.31.13:4629 , piece_id=SOTAFTXY3WGKKFB5WY4KQX7TH3NQ56X2OOHLDZ4I33ZP7SAVID4A, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:09.014143083 (536h55m8.885226128s) [noise=true , address=45.91.116.13:4156 , piece_id=UK6PAHAPXAEJ3TPZ6NMEMSQGQAGL6UHKDBY6DLVZRW6FSFZJU4ZA, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:09.032387904 (536h55m8.903470939s) [noise=true , address=45.91.116.14:4190 , piece_id=S2XD7GJQK2YKBD2W7ESEWK6ABRQTUZJSH5FXFIRV3QCDI3LHA6BA, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:09.261923207 (536h55m9.133006252s) [noise=true , address=45.91.116.4:4937 , piece_id=2YSWMFDVAPEL4RZVKLA2S77VL2RRWYSNC57VCCLYGU55ICDJWZBQ, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:09.308402360 (536h55m9.179485396s) [noise=true , address=77.90.31.16:4736 , piece_id=VKGQU254DOXYQZNTM6I2P6NGK3MC4YV3PALZJSCLRPSRLVKJII4A, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:09.330736548 (536h55m9.201819583s) [noise=true , address=45.91.116.12:4226 , piece_id=YGIUUZ6MIM6AQRGQTBXTIEHUIGXNFFWL456MMW36L567REVI5GZA, seg_pos={0 0}, upload=470877]: Done uploading piece. err: <nil> Dec 28 00:24:09.330755204 (536h55m9.201838239s) [seg_pos={0 0}, upload=470877]: Segment reached optimal threshold of 13 pieces. Dec 28 00:24:09.330925348 (536h55m9.202008383s) [noise=true , address=145.79.227.49:4337 , piece_id=WFDMFSR4PBEPBWA2HRTSXUEM3DNKQ3NIWKIHODXRKHQDLORSGB2A, seg_pos={0 0}, upload=470877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 28 00:24:09.330931319 (536h55m9.202014354s) [noise=true , address=145.79.227.49:4337 , piece_id=WFDMFSR4PBEPBWA2HRTSXUEM3DNKQ3NIWKIHODXRKHQDLORSGB2A, seg_pos={0 0}, upload=470877]: Long tail cancellation due to optimal threshold Dec 28 00:24:09.331041048 (536h55m9.202124103s) [noise=true , address=145.79.227.33:4871 , piece_id=Y2N5VCAWGQDFR6OTM6FYIXNQKPIWB2FU3CZMA63TVNAX7GQO3NJA, seg_pos={0 0}, upload=470877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 28 00:24:09.331051708 (536h55m9.202134763s) [noise=true , address=145.79.227.33:4871 , piece_id=Y2N5VCAWGQDFR6OTM6FYIXNQKPIWB2FU3CZMA63TVNAX7GQO3NJA, seg_pos={0 0}, upload=470877]: Long tail cancellation due to optimal threshold Dec 28 00:24:09.331118606 (536h55m9.202201641s) [noise=true , address=45.91.116.10:4081 , piece_id=BIR7PSNHRU2UUSNEKMHEB52HX5EJATQ2OOJ6WRGLABRB5ZPSQI5Q, seg_pos={0 0}, upload=470877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:55734->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:55734->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55734->45.91.116.10:4081: use of closed network connection Dec 28 00:24:09.331124617 (536h55m9.202207652s) [noise=true , address=45.91.116.10:4081 , piece_id=BIR7PSNHRU2UUSNEKMHEB52HX5EJATQ2OOJ6WRGLABRB5ZPSQI5Q, seg_pos={0 0}, upload=470877]: Long tail cancellation due to optimal threshold Dec 28 00:24:09.331157499 (536h55m9.202240535s) [seg_pos={0 0}, upload=470877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:24:09.331256749 (536h55m9.202339784s) [upload=470877]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:24:09.331264032 (536h55m9.202347067s) [upload=470877]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:24:09.624302747 (536h55m9.495385792s) [upload=470877]: Done uploading segments... Dec 28 00:24:09.624327444 (536h55m9.495410479s) [upload=470877]: Upload finished. err: <nil> Duration: 30.003469739s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 28.030028054s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:24:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 28.030050457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.X.XX.X.X] duration: 30.003628995s trace: https://jaeger.dev.storj.io/trace/59add87fd51b7f25 FAIL (2025-12-28 00:23:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g segments... Dec 28 00:23:11.865513741 (536h54m11.736596776s) [upload=470861]: Got next segment. Inline: false Dec 28 00:23:11.865548067 (536h54m11.736631102s) [upload=470861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:23:11.865562775 (536h54m11.73664581s) [upload=470861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:23:12.109413947 (536h54m11.980497002s) [seg_pos={0 0}, upload=470861]: Begin upload segment... Dec 28 00:23:12.109466357 (536h54m11.980549402s) [seg_pos={0 0}, upload=470861]: Done begin upload segment. Dec 28 00:23:12.109500923 (536h54m11.980583968s) [noise=true , address=77.90.31.14:4664 , piece_id=G45HEXQZZQEK5MDTHMVYAEOSHBQMGBINWNGFXOETSE5F6YAG72OQ, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.109552721 (536h54m11.980635756s) [noise=true , address=145.79.227.15:4873 , piece_id=6KXX7PXUOB3NGMDX3XB3NWU36QZHM6A2LOAJ6HXMQRBU6BFCDYQQ, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.110067552 (536h54m11.981150597s) [noise=true , address=145.79.227.24:4583 , piece_id=SEPZBM7GVEGX7F26OOPKPPLC24YPOTRNNNFN7UYIXLNXA3IUJDQQ, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.110647346 (536h54m11.981730381s) [noise=true , address=145.79.227.45:4230 , piece_id=2AWNAEXCNLWQUBNCMUBJUR3MM5BGKNRYOX36WPZLVL5U22QLMNIA, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.110721477 (536h54m11.981804522s) [noise=true , address=145.79.227.50:4374 , piece_id=FROQLGYRY72D35QYYZCZOXPMEMQVBS2OUQ5GXUOWS5DQRE4QGBAQ, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.111149432 (536h54m11.982232477s) [noise=true , address=45.91.116.10:4082 , piece_id=TT4B5HU3W4QW5I6U3MOM5S6VLI5LVSGUZXYRPEXLUDMVKIQP5MMA, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.111180731 (536h54m11.982263776s) [noise=true , address=145.79.227.16:4909 , piece_id=C5TBGVDDX4L5N33OSHK3ZHHGNP6ZYWSHA5L3HXCSKCBBEAGER5ZA, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.111563069 (536h54m11.982646115s) [noise=true , address=45.91.116.4:4937 , piece_id=IVDI6WM6L2QDR5MW2XWIMDWAIAT6PWYUDISWKMHXCIRUXAA5CNZA, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.111590843 (536h54m11.982673888s) [noise=true , address=45.91.116.5:4299 , piece_id=AW7DZTTGFKEQI4U5GBMXDQ7VAYQRGA5PX4GL5XY5GUVOZPS4IE5A, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.111966799 (536h54m11.983049844s) [noise=true , address=45.91.116.8:4045 , piece_id=RDKSVWNAHQKVZWYJPCAS66QEEVZTW4KJADSLQFHF5B5QENKP7VMA, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.112002076 (536h54m11.983085121s) [noise=true , address=45.91.116.11:4118 , piece_id=3USKEHRHHSBO74NGGORVH2D4K5AWXKS2DF3Q5AA5RFUTUFU4ZEVQ, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.112347934 (536h54m11.98343098s) [noise=true , address=145.79.227.21:4473 , piece_id=LYGJBEFPKKK7M6S5TNVBLJHCOZUUQ6CE2YYY32HKB6QPAYEKYDFA, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.112380146 (536h54m11.983463181s) [noise=true , address=145.79.227.37:4566 , piece_id=VWYZAEJPAVJ5T2G532PYNR7CBDGS5GDOPRQWKSCHNB6SMHKCJZ7A, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.112761141 (536h54m11.983844176s) [noise=true , address=145.79.227.17:4971 , piece_id=BTSH3CJXYYGDXZNX47JDYRPT6XZMZ6NVDCXMLP5JE5B7HFDZ73TQ, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.112786569 (536h54m11.983869604s) [noise=true , address=145.79.227.11:4729 , piece_id=PFQN5RJ74LGVWAAJGGGNQXE234RBAO5BZ5XMXQUXRCRUAIXCB7SA, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:12.113174859 (536h54m11.984257904s) [noise=true , address=145.79.227.41:4085 , piece_id=Z7VUVW4YKS64JFZ4NMJEZYPA2MBVW5V6HQWKUISLVWN7PTRLD35A, seg_pos={0 0}, upload=470861]: Uploading piece... Dec 28 00:23:13.738188981 (536h54m13.609272026s) [upload=470861]: Next returned nil segment Dec 28 00:23:13.738203579 (536h54m13.609286624s) [upload=470861]: Waiting for error group managing segments... Dec 28 00:23:14.115251159 (536h54m13.986334204s) [noise=true , address=145.79.227.15:4873 , piece_id=6KXX7PXUOB3NGMDX3XB3NWU36QZHM6A2LOAJ6HXMQRBU6BFCDYQQ, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.116411358 (536h54m13.987494403s) [noise=true , address=145.79.227.16:4909 , piece_id=C5TBGVDDX4L5N33OSHK3ZHHGNP6ZYWSHA5L3HXCSKCBBEAGER5ZA, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.119464973 (536h54m13.990548018s) [noise=true , address=145.79.227.17:4971 , piece_id=BTSH3CJXYYGDXZNX47JDYRPT6XZMZ6NVDCXMLP5JE5B7HFDZ73TQ, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.124828957 (536h54m13.995912002s) [noise=true , address=145.79.227.11:4729 , piece_id=PFQN5RJ74LGVWAAJGGGNQXE234RBAO5BZ5XMXQUXRCRUAIXCB7SA, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.127611654 (536h54m13.9986947s) [noise=true , address=145.79.227.41:4085 , piece_id=Z7VUVW4YKS64JFZ4NMJEZYPA2MBVW5V6HQWKUISLVWN7PTRLD35A, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.129365745 (536h54m14.00044878s) [noise=true , address=145.79.227.21:4473 , piece_id=LYGJBEFPKKK7M6S5TNVBLJHCOZUUQ6CE2YYY32HKB6QPAYEKYDFA, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.133719435 (536h54m14.00480248s) [noise=true , address=145.79.227.24:4583 , piece_id=SEPZBM7GVEGX7F26OOPKPPLC24YPOTRNNNFN7UYIXLNXA3IUJDQQ, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.136699110 (536h54m14.007782155s) [noise=true , address=145.79.227.45:4230 , piece_id=2AWNAEXCNLWQUBNCMUBJUR3MM5BGKNRYOX36WPZLVL5U22QLMNIA, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.138616931 (536h54m14.009699976s) [noise=true , address=145.79.227.50:4374 , piece_id=FROQLGYRY72D35QYYZCZOXPMEMQVBS2OUQ5GXUOWS5DQRE4QGBAQ, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.186744885 (536h54m14.05782792s) [noise=true , address=145.79.227.37:4566 , piece_id=VWYZAEJPAVJ5T2G532PYNR7CBDGS5GDOPRQWKSCHNB6SMHKCJZ7A, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.278243664 (536h54m14.149326699s) [noise=true , address=77.90.31.14:4664 , piece_id=G45HEXQZZQEK5MDTHMVYAEOSHBQMGBINWNGFXOETSE5F6YAG72OQ, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.352653571 (536h54m14.223736617s) [noise=true , address=45.91.116.11:4118 , piece_id=3USKEHRHHSBO74NGGORVH2D4K5AWXKS2DF3Q5AA5RFUTUFU4ZEVQ, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.584807830 (536h54m14.455890875s) [noise=true , address=45.91.116.4:4937 , piece_id=IVDI6WM6L2QDR5MW2XWIMDWAIAT6PWYUDISWKMHXCIRUXAA5CNZA, seg_pos={0 0}, upload=470861]: Done uploading piece. err: <nil> Dec 28 00:23:14.584832237 (536h54m14.455915282s) [seg_pos={0 0}, upload=470861]: Segment reached optimal threshold of 13 pieces. Dec 28 00:23:14.585041776 (536h54m14.456124821s) [noise=true , address=45.91.116.10:4082 , piece_id=TT4B5HU3W4QW5I6U3MOM5S6VLI5LVSGUZXYRPEXLUDMVKIQP5MMA, seg_pos={0 0}, upload=470861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:36690->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:36690->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36690->45.91.116.10:4082: use of closed network connection Dec 28 00:23:14.585049510 (536h54m14.456132556s) [noise=true , address=45.91.116.10:4082 , piece_id=TT4B5HU3W4QW5I6U3MOM5S6VLI5LVSGUZXYRPEXLUDMVKIQP5MMA, seg_pos={0 0}, upload=470861]: Long tail cancellation due to optimal threshold Dec 28 00:23:14.585371503 (536h54m14.456454548s) [noise=true , address=45.91.116.8:4045 , piece_id=RDKSVWNAHQKVZWYJPCAS66QEEVZTW4KJADSLQFHF5B5QENKP7VMA, seg_pos={0 0}, upload=470861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:44394->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:44394->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44394->45.91.116.8:4045: use of closed network connection Dec 28 00:23:14.585381593 (536h54m14.456464638s) [noise=true , address=45.91.116.8:4045 , piece_id=RDKSVWNAHQKVZWYJPCAS66QEEVZTW4KJADSLQFHF5B5QENKP7VMA, seg_pos={0 0}, upload=470861]: Long tail cancellation due to optimal threshold Dec 28 00:23:14.585642730 (536h54m14.456725775s) [noise=true , address=45.91.116.5:4299 , piece_id=AW7DZTTGFKEQI4U5GBMXDQ7VAYQRGA5PX4GL5XY5GUVOZPS4IE5A, seg_pos={0 0}, upload=470861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 00:23:14.585652448 (536h54m14.456735493s) [noise=true , address=45.91.116.5:4299 , piece_id=AW7DZTTGFKEQI4U5GBMXDQ7VAYQRGA5PX4GL5XY5GUVOZPS4IE5A, seg_pos={0 0}, upload=470861]: Long tail cancellation due to optimal threshold Dec 28 00:23:14.585676124 (536h54m14.456759169s) [seg_pos={0 0}, upload=470861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:23:14.585762879 (536h54m14.456845924s) [upload=470861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:23:14.585771085 (536h54m14.45685412s) [upload=470861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:23:14.881379070 (536h54m14.752462106s) [upload=470861]: Done uploading segments... Dec 28 00:23:14.881398557 (536h54m14.752481592s) [upload=470861]: Upload finished. err: <nil> Duration: 30.003650766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 15.316319492s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:23:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 15.316349868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 16.523248762s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:22:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 16.523277458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.X.XX.X.] duration: 6.03992619s trace: https://jaeger.dev.storj.io/trace/20525eba0c4e6923 FAIL (2025-12-28 00:22:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: OM4T45SARTGADG364QX4ZG4GZIWPRMNEYSSRYPREPEWJ754DXKTQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: HHSG5C7DQQ7LAJARUIJTWS7PHIEJXKKQMMX5OHY3QURV5BW66POQ) context deadline exceeded ======LOGS====== Dec 28 00:21:35.619931023 (536h52m35.491014058s) [upload=470843]: Starting upload Dec 28 00:21:35.620005074 (536h52m35.491088119s) [upload=470843]: Done starting upload Dec 28 00:21:35.620093012 (536h52m35.491176047s) [upload=470843]: Uploading segments... Dec 28 00:21:35.620237857 (536h52m35.491320903s) [upload=470843]: Got next segment. Inline: false Dec 28 00:21:35.620297682 (536h52m35.491380717s) [upload=470843]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:21:35.620304464 (536h52m35.491387509s) [upload=470843]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:21:35.865880962 (536h52m35.736963997s) [seg_pos={0 0}, upload=470843]: Begin upload segment... Dec 28 00:21:35.865926700 (536h52m35.737009735s) [seg_pos={0 0}, upload=470843]: Done begin upload segment. Dec 28 00:21:35.865967096 (536h52m35.737050131s) [noise=true , address=77.90.31.19:4844 , piece_id=676OVBO6TK345F3V2EE7R4RQ76BNEJHE5AKCI62OUMNTJXYUJ4FA, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.866020938 (536h52m35.737103983s) [noise=true , address=145.79.227.30:4761 , piece_id=WSEDGCHUZ4SZ5BDHMV7WRIPG6FXXGPS6WL2CQS2HPPXGL27QR6WQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.866616753 (536h52m35.737699788s) [noise=true , address=145.79.227.31:4798 , piece_id=SSCMFNKR2C3TLU7VG4PIHSIZBGEXMIL5WG7ADCQVLSBLTESL76UQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.866929458 (536h52m35.738012493s) [noise=true , address=145.79.227.46:4602 , piece_id=E5IQOOSFNERFF4Z2LXD2MVH7KVA47DEKQAEW5CO2BUUWTVAHBAEQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.867210083 (536h52m35.738293128s) [noise=true , address=45.91.116.11:4119 , piece_id=YBZKQQKPCAS27GUV4AWSHG6TUMGI3GM7ZYLSUCVMKMNSF5FBBEWA, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.867635423 (536h52m35.738718478s) [noise=true , address=145.79.227.51:4410 , piece_id=E7CGF2XDLVOEF2DA4CO4DYWRJ5ZQJGOEFD2GQVXLJ5KUDC4KFXXQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.868095379 (536h52m35.739178424s) [noise=true , address=77.90.31.17:4770 , piece_id=S3B66ITBUOGF5WLGNWCRD6AKUYVAUBDNSS2FYFS2W6GJ3ZTHGGJA, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.868482406 (536h52m35.739565441s) [noise=true , address=77.90.31.13:4629 , piece_id=5AE3F6AGLESAMDINDSF5753PPCMQEPAAQAXJWLNWZZ7Z4I4GYUVQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.868904770 (536h52m35.739987815s) [noise=true , address=145.79.227.4:4185 , piece_id=NKVHOYIO2KUIKXRBRZ3AJGAMBOXPB4GSG7DX3RCET6DTVB6ZQZ2A, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.870199755 (536h52m35.741282801s) [noise=true , address=45.91.116.13:4153 , piece_id=OM4T45SARTGADG364QX4ZG4GZIWPRMNEYSSRYPREPEWJ754DXKTQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.870294556 (536h52m35.741377601s) [noise=true , address=145.79.227.14:4839 , piece_id=35VN7K56LENOPKBUKYEMVIFJHR73TVYLZWRAG3L4Y3YE33BR3VQQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.871481486 (536h52m35.742564571s) [noise=true , address=45.91.116.7:4010 , piece_id=HHSG5C7DQQ7LAJARUIJTWS7PHIEJXKKQMMX5OHY3QURV5BW66POQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.872093752 (536h52m35.743176797s) [noise=true , address=45.91.116.6:4974 , piece_id=4GXYIQZ6DIZA45LP5RTQ3IJ5WNBW6AHVAJPXXOJAHUP7YJ6HS3QA, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.873515699 (536h52m35.744598744s) [noise=true , address=45.91.116.9:4263 , piece_id=DFVGRKDRGTPS4OXI7KMZDVXBWJOFU4RIJS2HJUCMV7VM6QWWEZPQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.873720279 (536h52m35.744803314s) [noise=true , address=145.79.227.42:4123 , piece_id=EJENWGOKFAYNNS7NYCOAPYWSRDALUD3VKLKDHIMLCSI4JKF4OWQA, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.877391901 (536h52m35.748474936s) [noise=true , address=45.91.116.5:4298 , piece_id=4EBN7TGTD7ZCWUFSH5ZXMNK4RELRNAM5J3NQZYO4XDNUYBZLGL6Q, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:37.519650408 (536h52m37.390733443s) [upload=470843]: Next returned nil segment Dec 28 00:21:37.519665497 (536h52m37.390748542s) [upload=470843]: Waiting for error group managing segments... Dec 28 00:21:37.875430782 (536h52m37.746513817s) [noise=true , address=145.79.227.46:4602 , piece_id=E5IQOOSFNERFF4Z2LXD2MVH7KVA47DEKQAEW5CO2BUUWTVAHBAEQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:37.880897151 (536h52m37.751980186s) [noise=true , address=145.79.227.4:4185 , piece_id=NKVHOYIO2KUIKXRBRZ3AJGAMBOXPB4GSG7DX3RCET6DTVB6ZQZ2A, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:37.884120449 (536h52m37.755203494s) [noise=true , address=145.79.227.31:4798 , piece_id=SSCMFNKR2C3TLU7VG4PIHSIZBGEXMIL5WG7ADCQVLSBLTESL76UQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:37.891646809 (536h52m37.762729844s) [noise=true , address=145.79.227.14:4839 , piece_id=35VN7K56LENOPKBUKYEMVIFJHR73TVYLZWRAG3L4Y3YE33BR3VQQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:37.894784524 (536h52m37.765867559s) [noise=true , address=145.79.227.51:4410 , piece_id=E7CGF2XDLVOEF2DA4CO4DYWRJ5ZQJGOEFD2GQVXLJ5KUDC4KFXXQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:37.907833842 (536h52m37.778916887s) [noise=true , address=145.79.227.42:4123 , piece_id=EJENWGOKFAYNNS7NYCOAPYWSRDALUD3VKLKDHIMLCSI4JKF4OWQA, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.062119988 (536h52m37.933203023s) [noise=true , address=77.90.31.19:4844 , piece_id=676OVBO6TK345F3V2EE7R4RQ76BNEJHE5AKCI62OUMNTJXYUJ4FA, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.093085972 (536h52m37.964169017s) [noise=true , address=45.91.116.7:4010 , piece_id=HHSG5C7DQQ7LAJARUIJTWS7PHIEJXKKQMMX5OHY3QURV5BW66POQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.113313689 (536h52m37.984396734s) [noise=true , address=45.91.116.11:4119 , piece_id=YBZKQQKPCAS27GUV4AWSHG6TUMGI3GM7ZYLSUCVMKMNSF5FBBEWA, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.114457598 (536h52m37.985540633s) [noise=true , address=45.91.116.13:4153 , piece_id=OM4T45SARTGADG364QX4ZG4GZIWPRMNEYSSRYPREPEWJ754DXKTQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.120849789 (536h52m37.991932834s) [noise=true , address=45.91.116.9:4263 , piece_id=DFVGRKDRGTPS4OXI7KMZDVXBWJOFU4RIJS2HJUCMV7VM6QWWEZPQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.188039950 (536h52m38.059122985s) [noise=true , address=45.91.116.5:4298 , piece_id=4EBN7TGTD7ZCWUFSH5ZXMNK4RELRNAM5J3NQZYO4XDNUYBZLGL6Q, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.212672815 (536h52m38.08375585s) [noise=true , address=45.91.116.6:4974 , piece_id=4GXYIQZ6DIZA45LP5RTQ3IJ5WNBW6AHVAJPXXOJAHUP7YJ6HS3QA, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.212691841 (536h52m38.083774886s) [seg_pos={0 0}, upload=470843]: Segment reached optimal threshold of 13 pieces. Dec 28 00:21:38.212833702 (536h52m38.083916737s) [noise=true , address=77.90.31.17:4770 , piece_id=S3B66ITBUOGF5WLGNWCRD6AKUYVAUBDNSS2FYFS2W6GJ3ZTHGGJA, seg_pos={0 0}, upload=470843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 00:21:38.212839693 (536h52m38.083922738s) [noise=true , address=77.90.31.17:4770 , piece_id=S3B66ITBUOGF5WLGNWCRD6AKUYVAUBDNSS2FYFS2W6GJ3ZTHGGJA, seg_pos={0 0}, upload=470843]: Long tail cancellation due to optimal threshold Dec 28 00:21:38.214197238 (536h52m38.085280273s) [noise=true , address=77.90.31.13:4629 , piece_id=5AE3F6AGLESAMDINDSF5753PPCMQEPAAQAXJWLNWZZ7Z4I4GYUVQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 00:21:38.214205023 (536h52m38.085288058s) [noise=true , address=77.90.31.13:4629 , piece_id=5AE3F6AGLESAMDINDSF5753PPCMQEPAAQAXJWLNWZZ7Z4I4GYUVQ, seg_pos={0 0}, upload=470843]: Long tail cancellation due to optimal threshold Dec 28 00:21:38.214310614 (536h52m38.085393649s) [noise=true , address=145.79.227.30:4761 , piece_id=WSEDGCHUZ4SZ5BDHMV7WRIPG6FXXGPS6WL2CQS2HPPXGL27QR6WQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 00:21:38.214316004 (536h52m38.085399039s) [noise=true , address=145.79.227.30:4761 , piece_id=WSEDGCHUZ4SZ5BDHMV7WRIPG6FXXGPS6WL2CQS2HPPXGL27QR6WQ, seg_pos={0 0}, upload=470843]: Long tail cancellation due to optimal threshold Dec 28 00:21:38.214356711 (536h52m38.085439756s) [seg_pos={0 0}, upload=470843]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:21:38.214429490 (536h52m38.085512525s) [upload=470843]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:21:38.214433798 (536h52m38.085516833s) [upload=470843]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:21:38.503006939 (536h52m38.374089974s) [upload=470843]: Done uploading segments... Dec 28 00:21:38.503026496 (536h52m38.374109531s) [upload=470843]: Upload finished. err: <nil> Duration: 6.039944355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.X.XX.X] duration: 30.004222801s trace: https://jaeger.dev.storj.io/trace/20525eba0c4e6923 FAIL (2025-12-28 00:22:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: OM4T45SARTGADG364QX4ZG4GZIWPRMNEYSSRYPREPEWJ754DXKTQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: HHSG5C7DQQ7LAJARUIJTWS7PHIEJXKKQMMX5OHY3QURV5BW66POQ) context deadline exceeded ======LOGS====== Dec 28 00:21:35.619931023 (536h52m35.491014058s) [upload=470843]: Starting upload Dec 28 00:21:35.620005074 (536h52m35.491088119s) [upload=470843]: Done starting upload Dec 28 00:21:35.620093012 (536h52m35.491176047s) [upload=470843]: Uploading segments... Dec 28 00:21:35.620237857 (536h52m35.491320903s) [upload=470843]: Got next segment. Inline: false Dec 28 00:21:35.620297682 (536h52m35.491380717s) [upload=470843]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:21:35.620304464 (536h52m35.491387509s) [upload=470843]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:21:35.865880962 (536h52m35.736963997s) [seg_pos={0 0}, upload=470843]: Begin upload segment... Dec 28 00:21:35.865926700 (536h52m35.737009735s) [seg_pos={0 0}, upload=470843]: Done begin upload segment. Dec 28 00:21:35.865967096 (536h52m35.737050131s) [noise=true , address=77.90.31.19:4844 , piece_id=676OVBO6TK345F3V2EE7R4RQ76BNEJHE5AKCI62OUMNTJXYUJ4FA, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.866020938 (536h52m35.737103983s) [noise=true , address=145.79.227.30:4761 , piece_id=WSEDGCHUZ4SZ5BDHMV7WRIPG6FXXGPS6WL2CQS2HPPXGL27QR6WQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.866616753 (536h52m35.737699788s) [noise=true , address=145.79.227.31:4798 , piece_id=SSCMFNKR2C3TLU7VG4PIHSIZBGEXMIL5WG7ADCQVLSBLTESL76UQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.866929458 (536h52m35.738012493s) [noise=true , address=145.79.227.46:4602 , piece_id=E5IQOOSFNERFF4Z2LXD2MVH7KVA47DEKQAEW5CO2BUUWTVAHBAEQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.867210083 (536h52m35.738293128s) [noise=true , address=45.91.116.11:4119 , piece_id=YBZKQQKPCAS27GUV4AWSHG6TUMGI3GM7ZYLSUCVMKMNSF5FBBEWA, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.867635423 (536h52m35.738718478s) [noise=true , address=145.79.227.51:4410 , piece_id=E7CGF2XDLVOEF2DA4CO4DYWRJ5ZQJGOEFD2GQVXLJ5KUDC4KFXXQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.868095379 (536h52m35.739178424s) [noise=true , address=77.90.31.17:4770 , piece_id=S3B66ITBUOGF5WLGNWCRD6AKUYVAUBDNSS2FYFS2W6GJ3ZTHGGJA, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.868482406 (536h52m35.739565441s) [noise=true , address=77.90.31.13:4629 , piece_id=5AE3F6AGLESAMDINDSF5753PPCMQEPAAQAXJWLNWZZ7Z4I4GYUVQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.868904770 (536h52m35.739987815s) [noise=true , address=145.79.227.4:4185 , piece_id=NKVHOYIO2KUIKXRBRZ3AJGAMBOXPB4GSG7DX3RCET6DTVB6ZQZ2A, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.870199755 (536h52m35.741282801s) [noise=true , address=45.91.116.13:4153 , piece_id=OM4T45SARTGADG364QX4ZG4GZIWPRMNEYSSRYPREPEWJ754DXKTQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.870294556 (536h52m35.741377601s) [noise=true , address=145.79.227.14:4839 , piece_id=35VN7K56LENOPKBUKYEMVIFJHR73TVYLZWRAG3L4Y3YE33BR3VQQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.871481486 (536h52m35.742564571s) [noise=true , address=45.91.116.7:4010 , piece_id=HHSG5C7DQQ7LAJARUIJTWS7PHIEJXKKQMMX5OHY3QURV5BW66POQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.872093752 (536h52m35.743176797s) [noise=true , address=45.91.116.6:4974 , piece_id=4GXYIQZ6DIZA45LP5RTQ3IJ5WNBW6AHVAJPXXOJAHUP7YJ6HS3QA, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.873515699 (536h52m35.744598744s) [noise=true , address=45.91.116.9:4263 , piece_id=DFVGRKDRGTPS4OXI7KMZDVXBWJOFU4RIJS2HJUCMV7VM6QWWEZPQ, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.873720279 (536h52m35.744803314s) [noise=true , address=145.79.227.42:4123 , piece_id=EJENWGOKFAYNNS7NYCOAPYWSRDALUD3VKLKDHIMLCSI4JKF4OWQA, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:35.877391901 (536h52m35.748474936s) [noise=true , address=45.91.116.5:4298 , piece_id=4EBN7TGTD7ZCWUFSH5ZXMNK4RELRNAM5J3NQZYO4XDNUYBZLGL6Q, seg_pos={0 0}, upload=470843]: Uploading piece... Dec 28 00:21:37.519650408 (536h52m37.390733443s) [upload=470843]: Next returned nil segment Dec 28 00:21:37.519665497 (536h52m37.390748542s) [upload=470843]: Waiting for error group managing segments... Dec 28 00:21:37.875430782 (536h52m37.746513817s) [noise=true , address=145.79.227.46:4602 , piece_id=E5IQOOSFNERFF4Z2LXD2MVH7KVA47DEKQAEW5CO2BUUWTVAHBAEQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:37.880897151 (536h52m37.751980186s) [noise=true , address=145.79.227.4:4185 , piece_id=NKVHOYIO2KUIKXRBRZ3AJGAMBOXPB4GSG7DX3RCET6DTVB6ZQZ2A, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:37.884120449 (536h52m37.755203494s) [noise=true , address=145.79.227.31:4798 , piece_id=SSCMFNKR2C3TLU7VG4PIHSIZBGEXMIL5WG7ADCQVLSBLTESL76UQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:37.891646809 (536h52m37.762729844s) [noise=true , address=145.79.227.14:4839 , piece_id=35VN7K56LENOPKBUKYEMVIFJHR73TVYLZWRAG3L4Y3YE33BR3VQQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:37.894784524 (536h52m37.765867559s) [noise=true , address=145.79.227.51:4410 , piece_id=E7CGF2XDLVOEF2DA4CO4DYWRJ5ZQJGOEFD2GQVXLJ5KUDC4KFXXQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:37.907833842 (536h52m37.778916887s) [noise=true , address=145.79.227.42:4123 , piece_id=EJENWGOKFAYNNS7NYCOAPYWSRDALUD3VKLKDHIMLCSI4JKF4OWQA, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.062119988 (536h52m37.933203023s) [noise=true , address=77.90.31.19:4844 , piece_id=676OVBO6TK345F3V2EE7R4RQ76BNEJHE5AKCI62OUMNTJXYUJ4FA, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.093085972 (536h52m37.964169017s) [noise=true , address=45.91.116.7:4010 , piece_id=HHSG5C7DQQ7LAJARUIJTWS7PHIEJXKKQMMX5OHY3QURV5BW66POQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.113313689 (536h52m37.984396734s) [noise=true , address=45.91.116.11:4119 , piece_id=YBZKQQKPCAS27GUV4AWSHG6TUMGI3GM7ZYLSUCVMKMNSF5FBBEWA, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.114457598 (536h52m37.985540633s) [noise=true , address=45.91.116.13:4153 , piece_id=OM4T45SARTGADG364QX4ZG4GZIWPRMNEYSSRYPREPEWJ754DXKTQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.120849789 (536h52m37.991932834s) [noise=true , address=45.91.116.9:4263 , piece_id=DFVGRKDRGTPS4OXI7KMZDVXBWJOFU4RIJS2HJUCMV7VM6QWWEZPQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.188039950 (536h52m38.059122985s) [noise=true , address=45.91.116.5:4298 , piece_id=4EBN7TGTD7ZCWUFSH5ZXMNK4RELRNAM5J3NQZYO4XDNUYBZLGL6Q, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.212672815 (536h52m38.08375585s) [noise=true , address=45.91.116.6:4974 , piece_id=4GXYIQZ6DIZA45LP5RTQ3IJ5WNBW6AHVAJPXXOJAHUP7YJ6HS3QA, seg_pos={0 0}, upload=470843]: Done uploading piece. err: <nil> Dec 28 00:21:38.212691841 (536h52m38.083774886s) [seg_pos={0 0}, upload=470843]: Segment reached optimal threshold of 13 pieces. Dec 28 00:21:38.212833702 (536h52m38.083916737s) [noise=true , address=77.90.31.17:4770 , piece_id=S3B66ITBUOGF5WLGNWCRD6AKUYVAUBDNSS2FYFS2W6GJ3ZTHGGJA, seg_pos={0 0}, upload=470843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 00:21:38.212839693 (536h52m38.083922738s) [noise=true , address=77.90.31.17:4770 , piece_id=S3B66ITBUOGF5WLGNWCRD6AKUYVAUBDNSS2FYFS2W6GJ3ZTHGGJA, seg_pos={0 0}, upload=470843]: Long tail cancellation due to optimal threshold Dec 28 00:21:38.214197238 (536h52m38.085280273s) [noise=true , address=77.90.31.13:4629 , piece_id=5AE3F6AGLESAMDINDSF5753PPCMQEPAAQAXJWLNWZZ7Z4I4GYUVQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 00:21:38.214205023 (536h52m38.085288058s) [noise=true , address=77.90.31.13:4629 , piece_id=5AE3F6AGLESAMDINDSF5753PPCMQEPAAQAXJWLNWZZ7Z4I4GYUVQ, seg_pos={0 0}, upload=470843]: Long tail cancellation due to optimal threshold Dec 28 00:21:38.214310614 (536h52m38.085393649s) [noise=true , address=145.79.227.30:4761 , piece_id=WSEDGCHUZ4SZ5BDHMV7WRIPG6FXXGPS6WL2CQS2HPPXGL27QR6WQ, seg_pos={0 0}, upload=470843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 00:21:38.214316004 (536h52m38.085399039s) [noise=true , address=145.79.227.30:4761 , piece_id=WSEDGCHUZ4SZ5BDHMV7WRIPG6FXXGPS6WL2CQS2HPPXGL27QR6WQ, seg_pos={0 0}, upload=470843]: Long tail cancellation due to optimal threshold Dec 28 00:21:38.214356711 (536h52m38.085439756s) [seg_pos={0 0}, upload=470843]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:21:38.214429490 (536h52m38.085512525s) [upload=470843]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:21:38.214433798 (536h52m38.085516833s) [upload=470843]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:21:38.503006939 (536h52m38.374089974s) [upload=470843]: Done uploading segments... Dec 28 00:21:38.503026496 (536h52m38.374109531s) [upload=470843]: Upload finished. err: <nil> Duration: 30.004245483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 31.417374661s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:21:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 31.41738955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002360541s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:21:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 1m0.002383115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.X.XX.] duration: 7.51518901s trace: https://jaeger.dev.storj.io/trace/1387adccf391d4eb FAIL (2025-12-28 00:20:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: WWKO62QLDVECL3CGFNX5DO5EYZ6ZTEBL5OELB4FVUTABP3NPQFTQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: V2WSCB3ANT6GAUXEC2M7SKF325N37WCHBE7IACPR7AFOWI2YOPXQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: WY3PNG5XSB4QVPP43BTUURCT67VS7TL6D7NJOHQQSOHL5AKQOHNA) context deadline exceeded ======LOGS====== Dec 28 00:20:10.890749246 (536h51m10.761832291s) [upload=470823]: Starting upload Dec 28 00:20:10.890825742 (536h51m10.761908787s) [upload=470823]: Done starting upload Dec 28 00:20:10.891032756 (536h51m10.762115801s) [upload=470823]: Uploading segments... Dec 28 00:20:10.893059886 (536h51m10.764142921s) [upload=470823]: Got next segment. Inline: false Dec 28 00:20:10.893134217 (536h51m10.764217273s) [upload=470823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:20:10.893141521 (536h51m10.764224566s) [upload=470823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:20:11.135146068 (536h51m11.006229113s) [seg_pos={0 0}, upload=470823]: Begin upload segment... Dec 28 00:20:11.135205671 (536h51m11.006288716s) [seg_pos={0 0}, upload=470823]: Done begin upload segment. Dec 28 00:20:11.135247691 (536h51m11.006330736s) [noise=true , address=145.79.227.38:4978 , piece_id=EVHHFE3NVDEQE2VQZXNV5UONITGPBNWTSI72SWQGGA7BQYHNIWPQ, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.136368535 (536h51m11.00745158s) [noise=true , address=145.79.227.12:4767 , piece_id=PMDQSV6NNPGLC3VLRGD5ZVXUI7PWMNRDPM272YXN6NCG6BZ5ZI3Q, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.137029905 (536h51m11.00811295s) [noise=true , address=145.79.227.40:4049 , piece_id=FGKUXN6W3Y4SRBMU6BV72AAMV4E2PZNQ7PCWFHNSCZTVRL6BHL2Q, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.137345766 (536h51m11.008428811s) [noise=true , address=45.91.116.5:4300 , piece_id=FVPTTLQYF4FNEFCBUV2RD4MCGXIQKCQYQ2AWT5S2VS52NHZNEKKA, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.137523234 (536h51m11.008606279s) [noise=true , address=77.90.31.10:4519 , piece_id=V2WSCB3ANT6GAUXEC2M7SKF325N37WCHBE7IACPR7AFOWI2YOPXQ, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.137937272 (536h51m11.009020317s) [noise=true , address=45.91.116.12:4225 , piece_id=EV4TNZISVT2MANNLFMRQDSNKA5WVQET4NA67O7YGAI6IUVAZ4HDA, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.138078541 (536h51m11.009161586s) [noise=true , address=145.79.227.34:4906 , piece_id=DK7YXXE55KUY43QW5ZDASBCWHCOP6LRATKYMIP6DOCCED6HEH2IA, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.138576670 (536h51m11.009659716s) [noise=true , address=45.91.116.10:4084 , piece_id=WY3PNG5XSB4QVPP43BTUURCT67VS7TL6D7NJOHQQSOHL5AKQOHNA, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.138974217 (536h51m11.010057262s) [noise=true , address=145.79.227.4:4185 , piece_id=IPDXERPPMGMHFALAFKMKXRMCZVCTY3KS3KYSYELFPJH6SHI5WXVA, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.139496301 (536h51m11.010579346s) [noise=true , address=145.79.227.3:4151 , piece_id=373Z5C2ADWOQPVE53SRNIFUAI5IRGPIBK5PS5LKWJQVG6CUMV3AQ, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.141267885 (536h51m11.01235093s) [noise=true , address=145.79.227.42:4121 , piece_id=2MN4OVNILICQFOFGUMQWNJ6IVS3OC27GM3NSAERLPX37YDRDWC3Q, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.142034344 (536h51m11.013117389s) [noise=true , address=45.91.116.8:4046 , piece_id=NOZCN2GBL2APAB26KNLB6WUK44FWNHMAVPDJX2BI3JVTVTI4ZY2Q, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.142230969 (536h51m11.013314004s) [noise=true , address=77.90.31.16:4736 , piece_id=WWKO62QLDVECL3CGFNX5DO5EYZ6ZTEBL5OELB4FVUTABP3NPQFTQ, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.142787058 (536h51m11.013870093s) [noise=true , address=45.91.116.4:4940 , piece_id=YMADLNQ7SRKADAWWSMZJJBL6IC4CDWNP5CSVJU62FQ3PSXECCX4A, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.143339599 (536h51m11.014422644s) [noise=true , address=145.79.227.16:4910 , piece_id=ITQPBYUUSG4AEFSBMVSZLK45LL5J574QDSD3P72MWFBZRUX2LVCQ, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.145406215 (536h51m11.01648926s) [noise=true , address=145.79.227.11:4732 , piece_id=W3QWVAH2JIP7QCUKLZ2IZEALAQB3CIO4OWEDQTEKXCK5SHLTHR5Q, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:12.842243938 (536h51m12.713326973s) [upload=470823]: Next returned nil segment Dec 28 00:20:12.842263736 (536h51m12.713346771s) [upload=470823]: Waiting for error group managing segments... Dec 28 00:20:13.141178990 (536h51m13.012262035s) [noise=true , address=145.79.227.38:4978 , piece_id=EVHHFE3NVDEQE2VQZXNV5UONITGPBNWTSI72SWQGGA7BQYHNIWPQ, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.146510522 (536h51m13.017593567s) [noise=true , address=145.79.227.11:4732 , piece_id=W3QWVAH2JIP7QCUKLZ2IZEALAQB3CIO4OWEDQTEKXCK5SHLTHR5Q, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.151070805 (536h51m13.02215385s) [noise=true , address=145.79.227.12:4767 , piece_id=PMDQSV6NNPGLC3VLRGD5ZVXUI7PWMNRDPM272YXN6NCG6BZ5ZI3Q, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.151667171 (536h51m13.022750246s) [noise=true , address=145.79.227.42:4121 , piece_id=2MN4OVNILICQFOFGUMQWNJ6IVS3OC27GM3NSAERLPX37YDRDWC3Q, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.152837109 (536h51m13.023920144s) [noise=true , address=145.79.227.4:4185 , piece_id=IPDXERPPMGMHFALAFKMKXRMCZVCTY3KS3KYSYELFPJH6SHI5WXVA, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.192274435 (536h51m13.06335747s) [noise=true , address=145.79.227.16:4910 , piece_id=ITQPBYUUSG4AEFSBMVSZLK45LL5J574QDSD3P72MWFBZRUX2LVCQ, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.378654212 (536h51m13.249737247s) [noise=true , address=45.91.116.8:4046 , piece_id=NOZCN2GBL2APAB26KNLB6WUK44FWNHMAVPDJX2BI3JVTVTI4ZY2Q, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.417614949 (536h51m13.288697984s) [noise=true , address=45.91.116.5:4300 , piece_id=FVPTTLQYF4FNEFCBUV2RD4MCGXIQKCQYQ2AWT5S2VS52NHZNEKKA, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.427677971 (536h51m13.298761006s) [noise=true , address=77.90.31.16:4736 , piece_id=WWKO62QLDVECL3CGFNX5DO5EYZ6ZTEBL5OELB4FVUTABP3NPQFTQ, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.695178146 (536h51m13.566261191s) [noise=true , address=45.91.116.10:4084 , piece_id=WY3PNG5XSB4QVPP43BTUURCT67VS7TL6D7NJOHQQSOHL5AKQOHNA, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:14.009380430 (536h51m13.880463465s) [noise=true , address=77.90.31.10:4519 , piece_id=V2WSCB3ANT6GAUXEC2M7SKF325N37WCHBE7IACPR7AFOWI2YOPXQ, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:14.382628214 (536h51m14.253711249s) [noise=true , address=145.79.227.3:4151 , piece_id=373Z5C2ADWOQPVE53SRNIFUAI5IRGPIBK5PS5LKWJQVG6CUMV3AQ, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:14.385775738 (536h51m14.256858773s) [noise=true , address=45.91.116.4:4940 , piece_id=YMADLNQ7SRKADAWWSMZJJBL6IC4CDWNP5CSVJU62FQ3PSXECCX4A, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:14.385788992 (536h51m14.256872038s) [seg_pos={0 0}, upload=470823]: Segment reached optimal threshold of 13 pieces. Dec 28 00:20:14.386018770 (536h51m14.257101805s) [noise=true , address=145.79.227.40:4049 , piece_id=FGKUXN6W3Y4SRBMU6BV72AAMV4E2PZNQ7PCWFHNSCZTVRL6BHL2Q, seg_pos={0 0}, upload=470823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:14.386026074 (536h51m14.257109119s) [noise=true , address=145.79.227.40:4049 , piece_id=FGKUXN6W3Y4SRBMU6BV72AAMV4E2PZNQ7PCWFHNSCZTVRL6BHL2Q, seg_pos={0 0}, upload=470823]: Long tail cancellation due to optimal threshold Dec 28 00:20:14.386130192 (536h51m14.257213227s) [noise=true , address=45.91.116.12:4225 , piece_id=EV4TNZISVT2MANNLFMRQDSNKA5WVQET4NA67O7YGAI6IUVAZ4HDA, seg_pos={0 0}, upload=470823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:14.386138167 (536h51m14.257221212s) [noise=true , address=45.91.116.12:4225 , piece_id=EV4TNZISVT2MANNLFMRQDSNKA5WVQET4NA67O7YGAI6IUVAZ4HDA, seg_pos={0 0}, upload=470823]: Long tail cancellation due to optimal threshold Dec 28 00:20:14.386435364 (536h51m14.257518399s) [noise=true , address=145.79.227.34:4906 , piece_id=DK7YXXE55KUY43QW5ZDASBCWHCOP6LRATKYMIP6DOCCED6HEH2IA, seg_pos={0 0}, upload=470823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:14.386442577 (536h51m14.257525622s) [noise=true , address=145.79.227.34:4906 , piece_id=DK7YXXE55KUY43QW5ZDASBCWHCOP6LRATKYMIP6DOCCED6HEH2IA, seg_pos={0 0}, upload=470823]: Long tail cancellation due to optimal threshold Dec 28 00:20:14.386476482 (536h51m14.257559517s) [seg_pos={0 0}, upload=470823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:20:14.386586171 (536h51m14.257669206s) [upload=470823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:14.386595748 (536h51m14.257678814s) [upload=470823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:14.676401026 (536h51m14.547484101s) [upload=470823]: Done uploading segments... Dec 28 00:20:14.676427797 (536h51m14.547510842s) [upload=470823]: Upload finished. err: <nil> Duration: 7.515202305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.X.XX] duration: 30.003448499s trace: https://jaeger.dev.storj.io/trace/1387adccf391d4eb FAIL (2025-12-28 00:20:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: WWKO62QLDVECL3CGFNX5DO5EYZ6ZTEBL5OELB4FVUTABP3NPQFTQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: V2WSCB3ANT6GAUXEC2M7SKF325N37WCHBE7IACPR7AFOWI2YOPXQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: WY3PNG5XSB4QVPP43BTUURCT67VS7TL6D7NJOHQQSOHL5AKQOHNA) context deadline exceeded ======LOGS====== Dec 28 00:20:10.890749246 (536h51m10.761832291s) [upload=470823]: Starting upload Dec 28 00:20:10.890825742 (536h51m10.761908787s) [upload=470823]: Done starting upload Dec 28 00:20:10.891032756 (536h51m10.762115801s) [upload=470823]: Uploading segments... Dec 28 00:20:10.893059886 (536h51m10.764142921s) [upload=470823]: Got next segment. Inline: false Dec 28 00:20:10.893134217 (536h51m10.764217273s) [upload=470823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:20:10.893141521 (536h51m10.764224566s) [upload=470823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:20:11.135146068 (536h51m11.006229113s) [seg_pos={0 0}, upload=470823]: Begin upload segment... Dec 28 00:20:11.135205671 (536h51m11.006288716s) [seg_pos={0 0}, upload=470823]: Done begin upload segment. Dec 28 00:20:11.135247691 (536h51m11.006330736s) [noise=true , address=145.79.227.38:4978 , piece_id=EVHHFE3NVDEQE2VQZXNV5UONITGPBNWTSI72SWQGGA7BQYHNIWPQ, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.136368535 (536h51m11.00745158s) [noise=true , address=145.79.227.12:4767 , piece_id=PMDQSV6NNPGLC3VLRGD5ZVXUI7PWMNRDPM272YXN6NCG6BZ5ZI3Q, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.137029905 (536h51m11.00811295s) [noise=true , address=145.79.227.40:4049 , piece_id=FGKUXN6W3Y4SRBMU6BV72AAMV4E2PZNQ7PCWFHNSCZTVRL6BHL2Q, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.137345766 (536h51m11.008428811s) [noise=true , address=45.91.116.5:4300 , piece_id=FVPTTLQYF4FNEFCBUV2RD4MCGXIQKCQYQ2AWT5S2VS52NHZNEKKA, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.137523234 (536h51m11.008606279s) [noise=true , address=77.90.31.10:4519 , piece_id=V2WSCB3ANT6GAUXEC2M7SKF325N37WCHBE7IACPR7AFOWI2YOPXQ, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.137937272 (536h51m11.009020317s) [noise=true , address=45.91.116.12:4225 , piece_id=EV4TNZISVT2MANNLFMRQDSNKA5WVQET4NA67O7YGAI6IUVAZ4HDA, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.138078541 (536h51m11.009161586s) [noise=true , address=145.79.227.34:4906 , piece_id=DK7YXXE55KUY43QW5ZDASBCWHCOP6LRATKYMIP6DOCCED6HEH2IA, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.138576670 (536h51m11.009659716s) [noise=true , address=45.91.116.10:4084 , piece_id=WY3PNG5XSB4QVPP43BTUURCT67VS7TL6D7NJOHQQSOHL5AKQOHNA, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.138974217 (536h51m11.010057262s) [noise=true , address=145.79.227.4:4185 , piece_id=IPDXERPPMGMHFALAFKMKXRMCZVCTY3KS3KYSYELFPJH6SHI5WXVA, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.139496301 (536h51m11.010579346s) [noise=true , address=145.79.227.3:4151 , piece_id=373Z5C2ADWOQPVE53SRNIFUAI5IRGPIBK5PS5LKWJQVG6CUMV3AQ, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.141267885 (536h51m11.01235093s) [noise=true , address=145.79.227.42:4121 , piece_id=2MN4OVNILICQFOFGUMQWNJ6IVS3OC27GM3NSAERLPX37YDRDWC3Q, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.142034344 (536h51m11.013117389s) [noise=true , address=45.91.116.8:4046 , piece_id=NOZCN2GBL2APAB26KNLB6WUK44FWNHMAVPDJX2BI3JVTVTI4ZY2Q, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.142230969 (536h51m11.013314004s) [noise=true , address=77.90.31.16:4736 , piece_id=WWKO62QLDVECL3CGFNX5DO5EYZ6ZTEBL5OELB4FVUTABP3NPQFTQ, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.142787058 (536h51m11.013870093s) [noise=true , address=45.91.116.4:4940 , piece_id=YMADLNQ7SRKADAWWSMZJJBL6IC4CDWNP5CSVJU62FQ3PSXECCX4A, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.143339599 (536h51m11.014422644s) [noise=true , address=145.79.227.16:4910 , piece_id=ITQPBYUUSG4AEFSBMVSZLK45LL5J574QDSD3P72MWFBZRUX2LVCQ, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:11.145406215 (536h51m11.01648926s) [noise=true , address=145.79.227.11:4732 , piece_id=W3QWVAH2JIP7QCUKLZ2IZEALAQB3CIO4OWEDQTEKXCK5SHLTHR5Q, seg_pos={0 0}, upload=470823]: Uploading piece... Dec 28 00:20:12.842243938 (536h51m12.713326973s) [upload=470823]: Next returned nil segment Dec 28 00:20:12.842263736 (536h51m12.713346771s) [upload=470823]: Waiting for error group managing segments... Dec 28 00:20:13.141178990 (536h51m13.012262035s) [noise=true , address=145.79.227.38:4978 , piece_id=EVHHFE3NVDEQE2VQZXNV5UONITGPBNWTSI72SWQGGA7BQYHNIWPQ, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.146510522 (536h51m13.017593567s) [noise=true , address=145.79.227.11:4732 , piece_id=W3QWVAH2JIP7QCUKLZ2IZEALAQB3CIO4OWEDQTEKXCK5SHLTHR5Q, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.151070805 (536h51m13.02215385s) [noise=true , address=145.79.227.12:4767 , piece_id=PMDQSV6NNPGLC3VLRGD5ZVXUI7PWMNRDPM272YXN6NCG6BZ5ZI3Q, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.151667171 (536h51m13.022750246s) [noise=true , address=145.79.227.42:4121 , piece_id=2MN4OVNILICQFOFGUMQWNJ6IVS3OC27GM3NSAERLPX37YDRDWC3Q, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.152837109 (536h51m13.023920144s) [noise=true , address=145.79.227.4:4185 , piece_id=IPDXERPPMGMHFALAFKMKXRMCZVCTY3KS3KYSYELFPJH6SHI5WXVA, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.192274435 (536h51m13.06335747s) [noise=true , address=145.79.227.16:4910 , piece_id=ITQPBYUUSG4AEFSBMVSZLK45LL5J574QDSD3P72MWFBZRUX2LVCQ, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.378654212 (536h51m13.249737247s) [noise=true , address=45.91.116.8:4046 , piece_id=NOZCN2GBL2APAB26KNLB6WUK44FWNHMAVPDJX2BI3JVTVTI4ZY2Q, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.417614949 (536h51m13.288697984s) [noise=true , address=45.91.116.5:4300 , piece_id=FVPTTLQYF4FNEFCBUV2RD4MCGXIQKCQYQ2AWT5S2VS52NHZNEKKA, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.427677971 (536h51m13.298761006s) [noise=true , address=77.90.31.16:4736 , piece_id=WWKO62QLDVECL3CGFNX5DO5EYZ6ZTEBL5OELB4FVUTABP3NPQFTQ, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:13.695178146 (536h51m13.566261191s) [noise=true , address=45.91.116.10:4084 , piece_id=WY3PNG5XSB4QVPP43BTUURCT67VS7TL6D7NJOHQQSOHL5AKQOHNA, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:14.009380430 (536h51m13.880463465s) [noise=true , address=77.90.31.10:4519 , piece_id=V2WSCB3ANT6GAUXEC2M7SKF325N37WCHBE7IACPR7AFOWI2YOPXQ, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:14.382628214 (536h51m14.253711249s) [noise=true , address=145.79.227.3:4151 , piece_id=373Z5C2ADWOQPVE53SRNIFUAI5IRGPIBK5PS5LKWJQVG6CUMV3AQ, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:14.385775738 (536h51m14.256858773s) [noise=true , address=45.91.116.4:4940 , piece_id=YMADLNQ7SRKADAWWSMZJJBL6IC4CDWNP5CSVJU62FQ3PSXECCX4A, seg_pos={0 0}, upload=470823]: Done uploading piece. err: <nil> Dec 28 00:20:14.385788992 (536h51m14.256872038s) [seg_pos={0 0}, upload=470823]: Segment reached optimal threshold of 13 pieces. Dec 28 00:20:14.386018770 (536h51m14.257101805s) [noise=true , address=145.79.227.40:4049 , piece_id=FGKUXN6W3Y4SRBMU6BV72AAMV4E2PZNQ7PCWFHNSCZTVRL6BHL2Q, seg_pos={0 0}, upload=470823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:14.386026074 (536h51m14.257109119s) [noise=true , address=145.79.227.40:4049 , piece_id=FGKUXN6W3Y4SRBMU6BV72AAMV4E2PZNQ7PCWFHNSCZTVRL6BHL2Q, seg_pos={0 0}, upload=470823]: Long tail cancellation due to optimal threshold Dec 28 00:20:14.386130192 (536h51m14.257213227s) [noise=true , address=45.91.116.12:4225 , piece_id=EV4TNZISVT2MANNLFMRQDSNKA5WVQET4NA67O7YGAI6IUVAZ4HDA, seg_pos={0 0}, upload=470823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:14.386138167 (536h51m14.257221212s) [noise=true , address=45.91.116.12:4225 , piece_id=EV4TNZISVT2MANNLFMRQDSNKA5WVQET4NA67O7YGAI6IUVAZ4HDA, seg_pos={0 0}, upload=470823]: Long tail cancellation due to optimal threshold Dec 28 00:20:14.386435364 (536h51m14.257518399s) [noise=true , address=145.79.227.34:4906 , piece_id=DK7YXXE55KUY43QW5ZDASBCWHCOP6LRATKYMIP6DOCCED6HEH2IA, seg_pos={0 0}, upload=470823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:14.386442577 (536h51m14.257525622s) [noise=true , address=145.79.227.34:4906 , piece_id=DK7YXXE55KUY43QW5ZDASBCWHCOP6LRATKYMIP6DOCCED6HEH2IA, seg_pos={0 0}, upload=470823]: Long tail cancellation due to optimal threshold Dec 28 00:20:14.386476482 (536h51m14.257559517s) [seg_pos={0 0}, upload=470823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:20:14.386586171 (536h51m14.257669206s) [upload=470823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:14.386595748 (536h51m14.257678814s) [upload=470823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:14.676401026 (536h51m14.547484101s) [upload=470823]: Done uploading segments... Dec 28 00:20:14.676427797 (536h51m14.547510842s) [upload=470823]: Upload finished. err: <nil> Duration: 30.003473418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.X.X] duration: 30.003513471s trace: https://jaeger.dev.storj.io/trace/78e18c7d1f52e238 FAIL (2025-12-28 00:19:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 00:19:11.509499103 (536h50m11.380582148s) [upload=470810]: Starting upload Dec 28 00:19:11.509559158 (536h50m11.380642203s) [upload=470810]: Done starting upload Dec 28 00:19:11.509698473 (536h50m11.380781508s) [upload=470810]: Uploading segments... Dec 28 00:19:11.509860641 (536h50m11.380943676s) [upload=470810]: Got next segment. Inline: false Dec 28 00:19:11.509908804 (536h50m11.380991839s) [upload=470810]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:19:11.509913893 (536h50m11.380996928s) [upload=470810]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:19:11.753468750 (536h50m11.624551795s) [seg_pos={0 0}, upload=470810]: Begin upload segment... Dec 28 00:19:11.753540927 (536h50m11.624623973s) [seg_pos={0 0}, upload=470810]: Done begin upload segment. Dec 28 00:19:11.753602495 (536h50m11.62468554s) [noise=true , address=77.90.31.10:4519 , piece_id=43OC2WKTBSLUYFKRSZKUXOMNOKLX3GIFG3UXF45SX625GLAX5FAA, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.753686885 (536h50m11.62476993s) [noise=true , address=145.79.227.37:4565 , piece_id=DHCCZ6SM5I4OXYWU6ZUEKTTRPMSPARCACYSGPI6A37EACNM3FNVA, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.754269815 (536h50m11.625352861s) [noise=true , address=145.79.227.50:4375 , piece_id=YATSRQXHGODSUAGZFQIKUXIIXD4WJYYGXQGEVOYABHYI3AE2MFAA, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.754766481 (536h50m11.625849516s) [noise=true , address=145.79.227.11:4731 , piece_id=AU6YT6AVZD2EQ4XAAOYLJUK2CMTGXAII2ELM3JQOB3VVSZ5DEUQA, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.755182784 (536h50m11.626265829s) [noise=true , address=145.79.227.13:4804 , piece_id=GFIRAR2Y33SZGBC6GDY47WQEBDHI5MJG55LAGIUNM4I2PLLCUDXA, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.755635686 (536h50m11.626718731s) [noise=true , address=45.91.116.5:4297 , piece_id=2NQVAVGEUWJCJ6AEKFEGYGNQX7DT3QJWTTNCWLVEJW2B3XRNPUIQ, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.755959133 (536h50m11.627042168s) [noise=true , address=45.91.116.4:4939 , piece_id=WJYCKD6WQUIECZM5HNOWDFJKYHVQNPPZZFN2UOGCGJWRSL2LQ4TA, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.756326321 (536h50m11.627409366s) [noise=true , address=145.79.227.5:4223 , piece_id=RNI6XR2HS4GZGQIWDNJ4XYCTBJQXKFGYWZ3MHGEW3TQ2V5TMI4RQ, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.756706004 (536h50m11.627789049s) [noise=true , address=77.90.31.15:4700 , piece_id=FNAXHWO44AEMWK3KDRINEGV5U4EHZZF5CCZEGHHSJQXEFFNXJGVQ, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.757774950 (536h50m11.628857985s) [noise=true , address=145.79.227.21:4473 , piece_id=YQ2QJA36Z3V23UFYJBDWFYGEPLVB3X5O5W6UDRT74AS2OAVMMSZA, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.759089714 (536h50m11.630172749s) [noise=true , address=45.91.116.8:4045 , piece_id=E4B2GJLKQW74VTCF2BQOFK3TNP3QLND56VDWX3KLMM3OM75P7HGQ, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.760486103 (536h50m11.631569148s) [noise=true , address=145.79.227.18:4007 , piece_id=5IJO7T34PUPU2SKHPKPIJFUULEK4BUVOOV5JP6FWZVBB64WBGGVQ, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.762561464 (536h50m11.633644499s) [noise=true , address=45.91.116.12:4226 , piece_id=GG6C2JLLN2SXMKZFDFYNNWFWU3Z56XRY5QJIA6XCRQUGY5ZSGYYA, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.763998050 (536h50m11.635081125s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYO4Q2NGTJI7UAU6V2MNTLIK2UNDE3PQA6YMSENQIH3YUNJHKQA, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.765746249 (536h50m11.636829294s) [noise=true , address=145.79.227.15:4873 , piece_id=BQEJUM4Z6TBLEMBK25W4WR4S23JMQ3SV5UGA3ZFBN22PCCOO66KA, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:11.766419010 (536h50m11.637502055s) [noise=true , address=77.90.31.19:4844 , piece_id=FXESG5M3AUEGID7S4PLJ3SQPYDSB4MVHEH2UN5ZSQ3BIHSHOQUFQ, seg_pos={0 0}, upload=470810]: Uploading piece... Dec 28 00:19:13.475103864 (536h50m13.346186899s) [upload=470810]: Next returned nil segment Dec 28 00:19:13.475125043 (536h50m13.346208088s) [upload=470810]: Waiting for error group managing segments... Dec 28 00:19:13.761133631 (536h50m13.632216666s) [noise=true , address=145.79.227.50:4375 , piece_id=YATSRQXHGODSUAGZFQIKUXIIXD4WJYYGXQGEVOYABHYI3AE2MFAA, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:13.762109960 (536h50m13.633193005s) [noise=true , address=145.79.227.5:4223 , piece_id=RNI6XR2HS4GZGQIWDNJ4XYCTBJQXKFGYWZ3MHGEW3TQ2V5TMI4RQ, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:13.763851366 (536h50m13.634934411s) [noise=true , address=145.79.227.15:4873 , piece_id=BQEJUM4Z6TBLEMBK25W4WR4S23JMQ3SV5UGA3ZFBN22PCCOO66KA, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:13.773588506 (536h50m13.644671551s) [noise=true , address=145.79.227.13:4804 , piece_id=GFIRAR2Y33SZGBC6GDY47WQEBDHI5MJG55LAGIUNM4I2PLLCUDXA, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:13.774235980 (536h50m13.645319025s) [noise=true , address=145.79.227.21:4473 , piece_id=YQ2QJA36Z3V23UFYJBDWFYGEPLVB3X5O5W6UDRT74AS2OAVMMSZA, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:13.785710909 (536h50m13.656793944s) [noise=true , address=145.79.227.37:4565 , piece_id=DHCCZ6SM5I4OXYWU6ZUEKTTRPMSPARCACYSGPI6A37EACNM3FNVA, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:13.787563898 (536h50m13.658646944s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYO4Q2NGTJI7UAU6V2MNTLIK2UNDE3PQA6YMSENQIH3YUNJHKQA, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:13.795373920 (536h50m13.666456955s) [noise=true , address=145.79.227.11:4731 , piece_id=AU6YT6AVZD2EQ4XAAOYLJUK2CMTGXAII2ELM3JQOB3VVSZ5DEUQA, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:13.798914121 (536h50m13.669997156s) [noise=true , address=145.79.227.18:4007 , piece_id=5IJO7T34PUPU2SKHPKPIJFUULEK4BUVOOV5JP6FWZVBB64WBGGVQ, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:14.030642609 (536h50m13.901725655s) [noise=true , address=77.90.31.10:4519 , piece_id=43OC2WKTBSLUYFKRSZKUXOMNOKLX3GIFG3UXF45SX625GLAX5FAA, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:14.063099913 (536h50m13.934182958s) [noise=true , address=77.90.31.15:4700 , piece_id=FNAXHWO44AEMWK3KDRINEGV5U4EHZZF5CCZEGHHSJQXEFFNXJGVQ, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:14.070815766 (536h50m13.941898841s) [noise=true , address=45.91.116.4:4939 , piece_id=WJYCKD6WQUIECZM5HNOWDFJKYHVQNPPZZFN2UOGCGJWRSL2LQ4TA, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:14.131351626 (536h50m14.002434671s) [noise=true , address=45.91.116.5:4297 , piece_id=2NQVAVGEUWJCJ6AEKFEGYGNQX7DT3QJWTTNCWLVEJW2B3XRNPUIQ, seg_pos={0 0}, upload=470810]: Done uploading piece. err: <nil> Dec 28 00:19:14.131375812 (536h50m14.002458857s) [seg_pos={0 0}, upload=470810]: Segment reached optimal threshold of 13 pieces. Dec 28 00:19:14.131606862 (536h50m14.002689918s) [noise=true , address=45.91.116.12:4226 , piece_id=GG6C2JLLN2SXMKZFDFYNNWFWU3Z56XRY5QJIA6XCRQUGY5ZSGYYA, seg_pos={0 0}, upload=470810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 00:19:14.131615980 (536h50m14.002699035s) [noise=true , address=45.91.116.12:4226 , piece_id=GG6C2JLLN2SXMKZFDFYNNWFWU3Z56XRY5QJIA6XCRQUGY5ZSGYYA, seg_pos={0 0}, upload=470810]: Long tail cancellation due to optimal threshold Dec 28 00:19:14.131991996 (536h50m14.003075041s) [noise=true , address=45.91.116.8:4045 , piece_id=E4B2GJLKQW74VTCF2BQOFK3TNP3QLND56VDWX3KLMM3OM75P7HGQ, seg_pos={0 0}, upload=470810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 00:19:14.132001063 (536h50m14.003084108s) [noise=true , address=45.91.116.8:4045 , piece_id=E4B2GJLKQW74VTCF2BQOFK3TNP3QLND56VDWX3KLMM3OM75P7HGQ, seg_pos={0 0}, upload=470810]: Long tail cancellation due to optimal threshold Dec 28 00:19:14.132135930 (536h50m14.003218975s) [noise=true , address=77.90.31.19:4844 , piece_id=FXESG5M3AUEGID7S4PLJ3SQPYDSB4MVHEH2UN5ZSQ3BIHSHOQUFQ, seg_pos={0 0}, upload=470810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:38786->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:38786->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38786->77.90.31.19:4844: use of closed network connection Dec 28 00:19:14.132142642 (536h50m14.003225677s) [noise=true , address=77.90.31.19:4844 , piece_id=FXESG5M3AUEGID7S4PLJ3SQPYDSB4MVHEH2UN5ZSQ3BIHSHOQUFQ, seg_pos={0 0}, upload=470810]: Long tail cancellation due to optimal threshold Dec 28 00:19:14.132166909 (536h50m14.003249944s) [seg_pos={0 0}, upload=470810]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:19:14.132276718 (536h50m14.003359763s) [upload=470810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:19:14.132282208 (536h50m14.003365253s) [upload=470810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:19:14.428345090 (536h50m14.299428135s) [upload=470810]: Done uploading segments... Dec 28 00:19:14.428368744 (536h50m14.299451789s) [upload=470810]: Upload finished. err: <nil> Duration: 30.00353895s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.477675642s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:19:15 UTC) orbiter warning: context deadline exceeded Duration: 3.477713435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.X.] duration: 29.943092877s trace: https://jaeger.dev.storj.io/trace/afbb874f5d1bd76 FAIL (2025-12-28 00:18:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: HKYSCIAXWCXG4VVLR4QXNDHIWGQNCN3XOXW3HCKXYDL3MMWXYGFA) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: RQJADNVBLRZNAZ2CYSCMTTMPDZ4GGIAAZGHGOZGXVJZ5RZBY6HUQ) context deadline exceeded ======LOGS====== segments... Dec 28 00:17:26.519682809 (536h48m26.390765874s) [upload=470784]: Got next segment. Inline: false Dec 28 00:17:26.519732393 (536h48m26.390815438s) [upload=470784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:17:26.519738766 (536h48m26.390929386s) [upload=470784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:17:26.767124128 (536h48m26.638207173s) [seg_pos={0 0}, upload=470784]: Begin upload segment... Dec 28 00:17:26.767182779 (536h48m26.638265824s) [seg_pos={0 0}, upload=470784]: Done begin upload segment. Dec 28 00:17:26.767221103 (536h48m26.638304148s) [noise=true , address=145.79.227.48:4302 , piece_id=PROJ3R6I7QOYSLXSA6FGM4RQJP3YOQJTAJPKZBV573ORBALG5MOQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.767425932 (536h48m26.638508977s) [noise=true , address=145.79.227.47:4265 , piece_id=EJ4ORP4YYKNF6VMO4IZJFTAADFT56BANLPUS2UOCPDN3ML6KD5CA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.768028951 (536h48m26.639111986s) [noise=true , address=145.79.227.38:4977 , piece_id=K7PI5CTLAEHJKTXRCL2D6OH2K6WKHJK23P372HXGRP7ALFC3NVRQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.768398855 (536h48m26.63948189s) [noise=true , address=77.90.31.14:4661 , piece_id=RQJADNVBLRZNAZ2CYSCMTTMPDZ4GGIAAZGHGOZGXVJZ5RZBY6HUQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.768675792 (536h48m26.639758827s) [noise=true , address=145.79.227.9:4365 , piece_id=GRI3TEEAES7TPWBU7QW5D5RVFF7BARNINEMDNKOKEEJH63Z7TXBQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.768948181 (536h48m26.640031216s) [noise=true , address=45.91.116.4:4940 , piece_id=UAT43AEPF4BVAWSBK4VIXARXUJEBP4KYPBLIT7JEHATXMR3BYPUA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.769195582 (536h48m26.640278627s) [noise=true , address=145.79.227.11:4732 , piece_id=26I734S6E2N6CPWXOB36H4V4TS6FQ5AWWMVFPSFYYE3SKEFQZY5A, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.769497227 (536h48m26.640580262s) [noise=true , address=145.79.227.15:4874 , piece_id=USTC2DNZKQX7OESCFQGUIEMCCLCMZRYKUGF3SQX6WAZWKFLQRYEA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.769853555 (536h48m26.6409366s) [noise=true , address=145.79.227.37:4566 , piece_id=NPWCR5KSU5NE3OV4XJQXXY3HZNM74Q4JLIJHTS2NQUV6HACL6RLA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.770130422 (536h48m26.641213467s) [noise=true , address=45.91.116.7:4012 , piece_id=NIRYXK6VCZKWWG3PYRXAY5D4LC6T7NSWQFYGHJPICHMQ4TUW7UGQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.770464509 (536h48m26.641547544s) [noise=true , address=45.91.116.13:4155 , piece_id=NJD7NVLSIY6VCPXXULEFLVX6QJ3VVANQ4TFPZ37XQWG2AA3JWM2Q, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.770908103 (536h48m26.641991138s) [noise=true , address=45.91.116.11:4117 , piece_id=HKYSCIAXWCXG4VVLR4QXNDHIWGQNCN3XOXW3HCKXYDL3MMWXYGFA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.771330257 (536h48m26.642413302s) [noise=true , address=45.91.116.6:4973 , piece_id=Y6WT4E7ITUCTR47OBPSWWS3W4CNJAQI4APPX5DEEVKBZXP7WJOUA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.772740522 (536h48m26.643823567s) [noise=true , address=145.79.227.41:4086 , piece_id=GB2EV47355FMVSWUALIRNFXCCPPM65CS2HYZBKWZGNSG3F63EVEQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.773173698 (536h48m26.644256733s) [noise=true , address=145.79.227.42:4122 , piece_id=4TZXAKO57ZG2DBDDYCCNRO2IWYQ7BBSSDYWWGHMSL63LN7EEZJGQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.773620618 (536h48m26.644703653s) [noise=true , address=145.79.227.17:4972 , piece_id=OSRA55NA5UZDOZH46SD6AZSHQQG6NPVQMGFHXNRZ2KZZ6NNKUVDQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:28.476242266 (536h48m28.347325302s) [upload=470784]: Next returned nil segment Dec 28 00:17:28.476257535 (536h48m28.34734057s) [upload=470784]: Waiting for error group managing segments... Dec 28 00:17:28.763256168 (536h48m28.634339213s) [noise=true , address=145.79.227.11:4732 , piece_id=26I734S6E2N6CPWXOB36H4V4TS6FQ5AWWMVFPSFYYE3SKEFQZY5A, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.766166960 (536h48m28.637250005s) [noise=true , address=145.79.227.42:4122 , piece_id=4TZXAKO57ZG2DBDDYCCNRO2IWYQ7BBSSDYWWGHMSL63LN7EEZJGQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.768789042 (536h48m28.639872087s) [noise=true , address=145.79.227.48:4302 , piece_id=PROJ3R6I7QOYSLXSA6FGM4RQJP3YOQJTAJPKZBV573ORBALG5MOQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.771419481 (536h48m28.642502516s) [noise=true , address=145.79.227.15:4874 , piece_id=USTC2DNZKQX7OESCFQGUIEMCCLCMZRYKUGF3SQX6WAZWKFLQRYEA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.780318858 (536h48m28.651401893s) [noise=true , address=145.79.227.17:4972 , piece_id=OSRA55NA5UZDOZH46SD6AZSHQQG6NPVQMGFHXNRZ2KZZ6NNKUVDQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.783358836 (536h48m28.654441871s) [noise=true , address=145.79.227.37:4566 , piece_id=NPWCR5KSU5NE3OV4XJQXXY3HZNM74Q4JLIJHTS2NQUV6HACL6RLA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.790105192 (536h48m28.661188227s) [noise=true , address=145.79.227.9:4365 , piece_id=GRI3TEEAES7TPWBU7QW5D5RVFF7BARNINEMDNKOKEEJH63Z7TXBQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.810964562 (536h48m28.682047597s) [noise=true , address=145.79.227.41:4086 , piece_id=GB2EV47355FMVSWUALIRNFXCCPPM65CS2HYZBKWZGNSG3F63EVEQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.820279549 (536h48m28.691362594s) [noise=true , address=145.79.227.38:4977 , piece_id=K7PI5CTLAEHJKTXRCL2D6OH2K6WKHJK23P372HXGRP7ALFC3NVRQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.938269961 (536h48m28.809352996s) [noise=true , address=77.90.31.14:4661 , piece_id=RQJADNVBLRZNAZ2CYSCMTTMPDZ4GGIAAZGHGOZGXVJZ5RZBY6HUQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:29.005507773 (536h48m28.876590818s) [noise=true , address=45.91.116.13:4155 , piece_id=NJD7NVLSIY6VCPXXULEFLVX6QJ3VVANQ4TFPZ37XQWG2AA3JWM2Q, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:29.029071322 (536h48m28.900154368s) [noise=true , address=45.91.116.11:4117 , piece_id=HKYSCIAXWCXG4VVLR4QXNDHIWGQNCN3XOXW3HCKXYDL3MMWXYGFA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:29.187091479 (536h48m29.058174524s) [noise=true , address=145.79.227.47:4265 , piece_id=EJ4ORP4YYKNF6VMO4IZJFTAADFT56BANLPUS2UOCPDN3ML6KD5CA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:29.187116656 (536h48m29.058199691s) [seg_pos={0 0}, upload=470784]: Segment reached optimal threshold of 13 pieces. Dec 28 00:17:29.187345462 (536h48m29.058428517s) [noise=true , address=45.91.116.6:4973 , piece_id=Y6WT4E7ITUCTR47OBPSWWS3W4CNJAQI4APPX5DEEVKBZXP7WJOUA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:54916->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:54916->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54916->45.91.116.6:4973: use of closed network connection Dec 28 00:17:29.187356713 (536h48m29.058439748s) [noise=true , address=45.91.116.6:4973 , piece_id=Y6WT4E7ITUCTR47OBPSWWS3W4CNJAQI4APPX5DEEVKBZXP7WJOUA, seg_pos={0 0}, upload=470784]: Long tail cancellation due to optimal threshold Dec 28 00:17:29.187405597 (536h48m29.058488632s) [noise=true , address=45.91.116.7:4012 , piece_id=NIRYXK6VCZKWWG3PYRXAY5D4LC6T7NSWQFYGHJPICHMQ4TUW7UGQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 00:17:29.187423440 (536h48m29.058506475s) [noise=true , address=45.91.116.7:4012 , piece_id=NIRYXK6VCZKWWG3PYRXAY5D4LC6T7NSWQFYGHJPICHMQ4TUW7UGQ, seg_pos={0 0}, upload=470784]: Long tail cancellation due to optimal threshold Dec 28 00:17:29.187632418 (536h48m29.058715453s) [noise=true , address=45.91.116.4:4940 , piece_id=UAT43AEPF4BVAWSBK4VIXARXUJEBP4KYPBLIT7JEHATXMR3BYPUA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:45638->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:45638->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45638->45.91.116.4:4940: use of closed network connection Dec 28 00:17:29.187639101 (536h48m29.058722136s) [noise=true , address=45.91.116.4:4940 , piece_id=UAT43AEPF4BVAWSBK4VIXARXUJEBP4KYPBLIT7JEHATXMR3BYPUA, seg_pos={0 0}, upload=470784]: Long tail cancellation due to optimal threshold Dec 28 00:17:29.187657947 (536h48m29.058740982s) [seg_pos={0 0}, upload=470784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:17:29.187727970 (536h48m29.058811005s) [upload=470784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:17:29.187733591 (536h48m29.058816626s) [upload=470784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:17:29.495719816 (536h48m29.366802852s) [upload=470784]: Done uploading segments... Dec 28 00:17:29.495739805 (536h48m29.36682284s) [upload=470784]: Upload finished. err: <nil> Duration: 29.943125298s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.433167108s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:18:19 UTC) orbiter warning: context deadline exceeded Duration: 3.433186896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.439363656s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:18:02 UTC) orbiter warning: context deadline exceeded Duration: 3.440189449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.X] duration: 30.002618285s trace: https://jaeger.dev.storj.io/trace/afbb874f5d1bd76 FAIL (2025-12-28 00:17:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: HKYSCIAXWCXG4VVLR4QXNDHIWGQNCN3XOXW3HCKXYDL3MMWXYGFA) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: RQJADNVBLRZNAZ2CYSCMTTMPDZ4GGIAAZGHGOZGXVJZ5RZBY6HUQ) context deadline exceeded ======LOGS====== segments... Dec 28 00:17:26.519682809 (536h48m26.390765874s) [upload=470784]: Got next segment. Inline: false Dec 28 00:17:26.519732393 (536h48m26.390815438s) [upload=470784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:17:26.519738766 (536h48m26.390929386s) [upload=470784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:17:26.767124128 (536h48m26.638207173s) [seg_pos={0 0}, upload=470784]: Begin upload segment... Dec 28 00:17:26.767182779 (536h48m26.638265824s) [seg_pos={0 0}, upload=470784]: Done begin upload segment. Dec 28 00:17:26.767221103 (536h48m26.638304148s) [noise=true , address=145.79.227.48:4302 , piece_id=PROJ3R6I7QOYSLXSA6FGM4RQJP3YOQJTAJPKZBV573ORBALG5MOQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.767425932 (536h48m26.638508977s) [noise=true , address=145.79.227.47:4265 , piece_id=EJ4ORP4YYKNF6VMO4IZJFTAADFT56BANLPUS2UOCPDN3ML6KD5CA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.768028951 (536h48m26.639111986s) [noise=true , address=145.79.227.38:4977 , piece_id=K7PI5CTLAEHJKTXRCL2D6OH2K6WKHJK23P372HXGRP7ALFC3NVRQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.768398855 (536h48m26.63948189s) [noise=true , address=77.90.31.14:4661 , piece_id=RQJADNVBLRZNAZ2CYSCMTTMPDZ4GGIAAZGHGOZGXVJZ5RZBY6HUQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.768675792 (536h48m26.639758827s) [noise=true , address=145.79.227.9:4365 , piece_id=GRI3TEEAES7TPWBU7QW5D5RVFF7BARNINEMDNKOKEEJH63Z7TXBQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.768948181 (536h48m26.640031216s) [noise=true , address=45.91.116.4:4940 , piece_id=UAT43AEPF4BVAWSBK4VIXARXUJEBP4KYPBLIT7JEHATXMR3BYPUA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.769195582 (536h48m26.640278627s) [noise=true , address=145.79.227.11:4732 , piece_id=26I734S6E2N6CPWXOB36H4V4TS6FQ5AWWMVFPSFYYE3SKEFQZY5A, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.769497227 (536h48m26.640580262s) [noise=true , address=145.79.227.15:4874 , piece_id=USTC2DNZKQX7OESCFQGUIEMCCLCMZRYKUGF3SQX6WAZWKFLQRYEA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.769853555 (536h48m26.6409366s) [noise=true , address=145.79.227.37:4566 , piece_id=NPWCR5KSU5NE3OV4XJQXXY3HZNM74Q4JLIJHTS2NQUV6HACL6RLA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.770130422 (536h48m26.641213467s) [noise=true , address=45.91.116.7:4012 , piece_id=NIRYXK6VCZKWWG3PYRXAY5D4LC6T7NSWQFYGHJPICHMQ4TUW7UGQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.770464509 (536h48m26.641547544s) [noise=true , address=45.91.116.13:4155 , piece_id=NJD7NVLSIY6VCPXXULEFLVX6QJ3VVANQ4TFPZ37XQWG2AA3JWM2Q, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.770908103 (536h48m26.641991138s) [noise=true , address=45.91.116.11:4117 , piece_id=HKYSCIAXWCXG4VVLR4QXNDHIWGQNCN3XOXW3HCKXYDL3MMWXYGFA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.771330257 (536h48m26.642413302s) [noise=true , address=45.91.116.6:4973 , piece_id=Y6WT4E7ITUCTR47OBPSWWS3W4CNJAQI4APPX5DEEVKBZXP7WJOUA, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.772740522 (536h48m26.643823567s) [noise=true , address=145.79.227.41:4086 , piece_id=GB2EV47355FMVSWUALIRNFXCCPPM65CS2HYZBKWZGNSG3F63EVEQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.773173698 (536h48m26.644256733s) [noise=true , address=145.79.227.42:4122 , piece_id=4TZXAKO57ZG2DBDDYCCNRO2IWYQ7BBSSDYWWGHMSL63LN7EEZJGQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:26.773620618 (536h48m26.644703653s) [noise=true , address=145.79.227.17:4972 , piece_id=OSRA55NA5UZDOZH46SD6AZSHQQG6NPVQMGFHXNRZ2KZZ6NNKUVDQ, seg_pos={0 0}, upload=470784]: Uploading piece... Dec 28 00:17:28.476242266 (536h48m28.347325302s) [upload=470784]: Next returned nil segment Dec 28 00:17:28.476257535 (536h48m28.34734057s) [upload=470784]: Waiting for error group managing segments... Dec 28 00:17:28.763256168 (536h48m28.634339213s) [noise=true , address=145.79.227.11:4732 , piece_id=26I734S6E2N6CPWXOB36H4V4TS6FQ5AWWMVFPSFYYE3SKEFQZY5A, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.766166960 (536h48m28.637250005s) [noise=true , address=145.79.227.42:4122 , piece_id=4TZXAKO57ZG2DBDDYCCNRO2IWYQ7BBSSDYWWGHMSL63LN7EEZJGQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.768789042 (536h48m28.639872087s) [noise=true , address=145.79.227.48:4302 , piece_id=PROJ3R6I7QOYSLXSA6FGM4RQJP3YOQJTAJPKZBV573ORBALG5MOQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.771419481 (536h48m28.642502516s) [noise=true , address=145.79.227.15:4874 , piece_id=USTC2DNZKQX7OESCFQGUIEMCCLCMZRYKUGF3SQX6WAZWKFLQRYEA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.780318858 (536h48m28.651401893s) [noise=true , address=145.79.227.17:4972 , piece_id=OSRA55NA5UZDOZH46SD6AZSHQQG6NPVQMGFHXNRZ2KZZ6NNKUVDQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.783358836 (536h48m28.654441871s) [noise=true , address=145.79.227.37:4566 , piece_id=NPWCR5KSU5NE3OV4XJQXXY3HZNM74Q4JLIJHTS2NQUV6HACL6RLA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.790105192 (536h48m28.661188227s) [noise=true , address=145.79.227.9:4365 , piece_id=GRI3TEEAES7TPWBU7QW5D5RVFF7BARNINEMDNKOKEEJH63Z7TXBQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.810964562 (536h48m28.682047597s) [noise=true , address=145.79.227.41:4086 , piece_id=GB2EV47355FMVSWUALIRNFXCCPPM65CS2HYZBKWZGNSG3F63EVEQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.820279549 (536h48m28.691362594s) [noise=true , address=145.79.227.38:4977 , piece_id=K7PI5CTLAEHJKTXRCL2D6OH2K6WKHJK23P372HXGRP7ALFC3NVRQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:28.938269961 (536h48m28.809352996s) [noise=true , address=77.90.31.14:4661 , piece_id=RQJADNVBLRZNAZ2CYSCMTTMPDZ4GGIAAZGHGOZGXVJZ5RZBY6HUQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:29.005507773 (536h48m28.876590818s) [noise=true , address=45.91.116.13:4155 , piece_id=NJD7NVLSIY6VCPXXULEFLVX6QJ3VVANQ4TFPZ37XQWG2AA3JWM2Q, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:29.029071322 (536h48m28.900154368s) [noise=true , address=45.91.116.11:4117 , piece_id=HKYSCIAXWCXG4VVLR4QXNDHIWGQNCN3XOXW3HCKXYDL3MMWXYGFA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:29.187091479 (536h48m29.058174524s) [noise=true , address=145.79.227.47:4265 , piece_id=EJ4ORP4YYKNF6VMO4IZJFTAADFT56BANLPUS2UOCPDN3ML6KD5CA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: <nil> Dec 28 00:17:29.187116656 (536h48m29.058199691s) [seg_pos={0 0}, upload=470784]: Segment reached optimal threshold of 13 pieces. Dec 28 00:17:29.187345462 (536h48m29.058428517s) [noise=true , address=45.91.116.6:4973 , piece_id=Y6WT4E7ITUCTR47OBPSWWS3W4CNJAQI4APPX5DEEVKBZXP7WJOUA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:54916->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:54916->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54916->45.91.116.6:4973: use of closed network connection Dec 28 00:17:29.187356713 (536h48m29.058439748s) [noise=true , address=45.91.116.6:4973 , piece_id=Y6WT4E7ITUCTR47OBPSWWS3W4CNJAQI4APPX5DEEVKBZXP7WJOUA, seg_pos={0 0}, upload=470784]: Long tail cancellation due to optimal threshold Dec 28 00:17:29.187405597 (536h48m29.058488632s) [noise=true , address=45.91.116.7:4012 , piece_id=NIRYXK6VCZKWWG3PYRXAY5D4LC6T7NSWQFYGHJPICHMQ4TUW7UGQ, seg_pos={0 0}, upload=470784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 00:17:29.187423440 (536h48m29.058506475s) [noise=true , address=45.91.116.7:4012 , piece_id=NIRYXK6VCZKWWG3PYRXAY5D4LC6T7NSWQFYGHJPICHMQ4TUW7UGQ, seg_pos={0 0}, upload=470784]: Long tail cancellation due to optimal threshold Dec 28 00:17:29.187632418 (536h48m29.058715453s) [noise=true , address=45.91.116.4:4940 , piece_id=UAT43AEPF4BVAWSBK4VIXARXUJEBP4KYPBLIT7JEHATXMR3BYPUA, seg_pos={0 0}, upload=470784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:45638->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:45638->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45638->45.91.116.4:4940: use of closed network connection Dec 28 00:17:29.187639101 (536h48m29.058722136s) [noise=true , address=45.91.116.4:4940 , piece_id=UAT43AEPF4BVAWSBK4VIXARXUJEBP4KYPBLIT7JEHATXMR3BYPUA, seg_pos={0 0}, upload=470784]: Long tail cancellation due to optimal threshold Dec 28 00:17:29.187657947 (536h48m29.058740982s) [seg_pos={0 0}, upload=470784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:17:29.187727970 (536h48m29.058811005s) [upload=470784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:17:29.187733591 (536h48m29.058816626s) [upload=470784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:17:29.495719816 (536h48m29.366802852s) [upload=470784]: Done uploading segments... Dec 28 00:17:29.495739805 (536h48m29.36682284s) [upload=470784]: Upload finished. err: <nil> Duration: 30.002631832s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.482772834s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:16:55 UTC) orbiter warning: context deadline exceeded Duration: 3.482830785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X.] duration: 5.997406567s trace: https://jaeger.dev.storj.io/trace/2716bf6b9d375dd8 FAIL (2025-12-28 00:16:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 4ESIJ3UZV6AU77DABUFVONFXNBMBPRJQGF6X4LSIBJ7OIXY7LU2Q) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: REYB2K655GSM763ZUN4VS7YH3S37T6JNRZJ4XWK6YZDEEB4NEW6Q) context deadline exceeded ======LOGS====== Dec 28 00:15:08.430733042 (536h46m8.301816087s) [upload=470746]: Starting upload Dec 28 00:15:08.430819006 (536h46m8.301902052s) [upload=470746]: Done starting upload Dec 28 00:15:08.430961518 (536h46m8.302044553s) [upload=470746]: Uploading segments... Dec 28 00:15:08.431036290 (536h46m8.302119335s) [upload=470746]: Got next segment. Inline: false Dec 28 00:15:08.431072028 (536h46m8.302155063s) [upload=470746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:15:08.431077598 (536h46m8.302160634s) [upload=470746]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:15:08.676602809 (536h46m8.547685854s) [seg_pos={0 0}, upload=470746]: Begin upload segment... Dec 28 00:15:08.676662713 (536h46m8.547745758s) [seg_pos={0 0}, upload=470746]: Done begin upload segment. Dec 28 00:15:08.676701146 (536h46m8.547784221s) [noise=true , address=77.90.31.16:4736 , piece_id=4ESIJ3UZV6AU77DABUFVONFXNBMBPRJQGF6X4LSIBJ7OIXY7LU2Q, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.676720973 (536h46m8.547804018s) [noise=true , address=145.79.227.52:4445 , piece_id=OLCOGVGMYPBQ3LDY2IS3X57UDYATY6FK5NU2JXGHCWPZ376YHSVA, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.676887721 (536h46m8.547970756s) [noise=true , address=145.79.227.49:4337 , piece_id=AKIGSSXRBM6MJQA7D2H6SKKT76DOUZSC5IPTYFSG6KARAW7BTWZQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.677303463 (536h46m8.548386508s) [noise=true , address=145.79.227.15:4873 , piece_id=2IVDPBLBJG7HWZ3I7XF64PL6CRRSWFWRUEO3BGPCYI3AILBW7SCQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.677314163 (536h46m8.548397198s) [noise=true , address=145.79.227.19:4078 , piece_id=GYOZXF6MXQX6CGPEJOV6KDSLOPUFQQ3ERP7V4XWMOKMW77CY73HA, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.677646366 (536h46m8.548729411s) [noise=true , address=145.79.227.31:4798 , piece_id=7Q2W25GRLDDQSLTXRBWRZ6JM7CKTXUEHJE4VTVRR33GREAXHQV3A, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.677652608 (536h46m8.548735653s) [noise=true , address=145.79.227.32:4836 , piece_id=V3BVXMBTCW3IA5EUBMAXNRKJ62KSTEHCZVXGCV5HQTNQ2FMQRWLQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.677972827 (536h46m8.549055862s) [noise=true , address=45.91.116.4:4940 , piece_id=2YIJZXP2EDO3ZNSLG7BXMKKOWLSLQBQ7IZIROQYMHAJM43WGR2SA, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.678032150 (536h46m8.549115185s) [noise=true , address=77.90.31.12:4590 , piece_id=REYB2K655GSM763ZUN4VS7YH3S37T6JNRZJ4XWK6YZDEEB4NEW6Q, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.678660537 (536h46m8.549743612s) [noise=true , address=45.91.116.9:4261 , piece_id=TGY3V36OS4V7SLFNCVU655EJHTV4INGJHWBYXE6TFCICP2TTLLKQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.679059757 (536h46m8.550142802s) [noise=true , address=145.79.227.37:4566 , piece_id=2MBITVMRJPF5OQESRNLV6DI2TEXF5DY7CAFQRKEX6SQHOVCWSMEA, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.679379977 (536h46m8.550463022s) [noise=true , address=145.79.227.41:4086 , piece_id=HAZ5Q5NIIYTSLU2YG6BWOPMIQ2REYGHTFHJMVMIXUJFKCDRC42KA, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.679587512 (536h46m8.550670557s) [noise=true , address=145.79.227.42:4122 , piece_id=EGOB7OL6SHQZXE7DYJQYZBME6QOFKPUKI6Y3YLKNMJCKBI3447PQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.679744451 (536h46m8.550827486s) [noise=true , address=145.79.227.51:4409 , piece_id=VUBKBPPODYQF2FS2NS3JQDZSIS5EAVBDG3BSIBXOEOS473KB4WTQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.680028902 (536h46m8.551111947s) [noise=true , address=145.79.227.46:4603 , piece_id=YH7VMT63SCTAUET4ZL434DHXZAD4VJWIK26KFQRFMLIWUDVIH6TQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.680701944 (536h46m8.551784979s) [noise=true , address=145.79.227.14:4839 , piece_id=RDO3EXV4O6XMBJCCBDYKBNBSH5JPKMQX24HYVP6OZR75F4KAHO2A, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:10.383427059 (536h46m10.254510094s) [upload=470746]: Next returned nil segment Dec 28 00:15:10.383446646 (536h46m10.254529691s) [upload=470746]: Waiting for error group managing segments... Dec 28 00:15:10.670104790 (536h46m10.541187835s) [noise=true , address=145.79.227.32:4836 , piece_id=V3BVXMBTCW3IA5EUBMAXNRKJ62KSTEHCZVXGCV5HQTNQ2FMQRWLQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.678994417 (536h46m10.550077452s) [noise=true , address=145.79.227.52:4445 , piece_id=OLCOGVGMYPBQ3LDY2IS3X57UDYATY6FK5NU2JXGHCWPZ376YHSVA, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.679846740 (536h46m10.550929785s) [noise=true , address=145.79.227.46:4603 , piece_id=YH7VMT63SCTAUET4ZL434DHXZAD4VJWIK26KFQRFMLIWUDVIH6TQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.683718673 (536h46m10.554801718s) [noise=true , address=145.79.227.37:4566 , piece_id=2MBITVMRJPF5OQESRNLV6DI2TEXF5DY7CAFQRKEX6SQHOVCWSMEA, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.684543243 (536h46m10.555626278s) [noise=true , address=145.79.227.15:4873 , piece_id=2IVDPBLBJG7HWZ3I7XF64PL6CRRSWFWRUEO3BGPCYI3AILBW7SCQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.685269356 (536h46m10.556352401s) [noise=true , address=145.79.227.42:4122 , piece_id=EGOB7OL6SHQZXE7DYJQYZBME6QOFKPUKI6Y3YLKNMJCKBI3447PQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.690006967 (536h46m10.561090012s) [noise=true , address=145.79.227.31:4798 , piece_id=7Q2W25GRLDDQSLTXRBWRZ6JM7CKTXUEHJE4VTVRR33GREAXHQV3A, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.697989727 (536h46m10.569072773s) [noise=true , address=145.79.227.19:4078 , piece_id=GYOZXF6MXQX6CGPEJOV6KDSLOPUFQQ3ERP7V4XWMOKMW77CY73HA, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.699565818 (536h46m10.570648863s) [noise=true , address=145.79.227.51:4409 , piece_id=VUBKBPPODYQF2FS2NS3JQDZSIS5EAVBDG3BSIBXOEOS473KB4WTQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.718776609 (536h46m10.589859654s) [noise=true , address=145.79.227.14:4839 , piece_id=RDO3EXV4O6XMBJCCBDYKBNBSH5JPKMQX24HYVP6OZR75F4KAHO2A, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.908261129 (536h46m10.779344174s) [noise=true , address=77.90.31.12:4590 , piece_id=REYB2K655GSM763ZUN4VS7YH3S37T6JNRZJ4XWK6YZDEEB4NEW6Q, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.986794920 (536h46m10.857877955s) [noise=true , address=77.90.31.16:4736 , piece_id=4ESIJ3UZV6AU77DABUFVONFXNBMBPRJQGF6X4LSIBJ7OIXY7LU2Q, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:11.001995684 (536h46m10.873078729s) [noise=true , address=45.91.116.9:4261 , piece_id=TGY3V36OS4V7SLFNCVU655EJHTV4INGJHWBYXE6TFCICP2TTLLKQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:11.002018358 (536h46m10.873101393s) [seg_pos={0 0}, upload=470746]: Segment reached optimal threshold of 13 pieces. Dec 28 00:15:11.002062251 (536h46m10.873145296s) [noise=true , address=45.91.116.4:4940 , piece_id=2YIJZXP2EDO3ZNSLG7BXMKKOWLSLQBQ7IZIROQYMHAJM43WGR2SA, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:11.002247834 (536h46m10.873330879s) [noise=true , address=145.79.227.49:4337 , piece_id=AKIGSSXRBM6MJQA7D2H6SKKT76DOUZSC5IPTYFSG6KARAW7BTWZQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 28 00:15:11.002256061 (536h46m10.873339106s) [noise=true , address=145.79.227.49:4337 , piece_id=AKIGSSXRBM6MJQA7D2H6SKKT76DOUZSC5IPTYFSG6KARAW7BTWZQ, seg_pos={0 0}, upload=470746]: Long tail cancellation due to optimal threshold Dec 28 00:15:11.002536735 (536h46m10.87361978s) [noise=true , address=145.79.227.41:4086 , piece_id=HAZ5Q5NIIYTSLU2YG6BWOPMIQ2REYGHTFHJMVMIXUJFKCDRC42KA, seg_pos={0 0}, upload=470746]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34508->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34508->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34508->145.79.227.41:4086: use of closed network connection Dec 28 00:15:11.002548447 (536h46m10.873631492s) [noise=true , address=145.79.227.41:4086 , piece_id=HAZ5Q5NIIYTSLU2YG6BWOPMIQ2REYGHTFHJMVMIXUJFKCDRC42KA, seg_pos={0 0}, upload=470746]: Long tail cancellation due to optimal threshold Dec 28 00:15:11.002573375 (536h46m10.87365641s) [seg_pos={0 0}, upload=470746]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 00:15:11.002678515 (536h46m10.87376155s) [upload=470746]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:15:11.002687191 (536h46m10.873770236s) [upload=470746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:15:11.289509848 (536h46m11.160592904s) [upload=470746]: Done uploading segments... Dec 28 00:15:11.289541920 (536h46m11.160624965s) [upload=470746]: Upload finished. err: <nil> Duration: 5.997432788s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.523339482s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:16:15 UTC) orbiter warning: context deadline exceeded Duration: 3.523395449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.X] duration: 30.002358295s trace: https://jaeger.dev.storj.io/trace/2716bf6b9d375dd8 FAIL (2025-12-28 00:15:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 4ESIJ3UZV6AU77DABUFVONFXNBMBPRJQGF6X4LSIBJ7OIXY7LU2Q) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: REYB2K655GSM763ZUN4VS7YH3S37T6JNRZJ4XWK6YZDEEB4NEW6Q) context deadline exceeded ======LOGS====== Dec 28 00:15:08.430733042 (536h46m8.301816087s) [upload=470746]: Starting upload Dec 28 00:15:08.430819006 (536h46m8.301902052s) [upload=470746]: Done starting upload Dec 28 00:15:08.430961518 (536h46m8.302044553s) [upload=470746]: Uploading segments... Dec 28 00:15:08.431036290 (536h46m8.302119335s) [upload=470746]: Got next segment. Inline: false Dec 28 00:15:08.431072028 (536h46m8.302155063s) [upload=470746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:15:08.431077598 (536h46m8.302160634s) [upload=470746]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:15:08.676602809 (536h46m8.547685854s) [seg_pos={0 0}, upload=470746]: Begin upload segment... Dec 28 00:15:08.676662713 (536h46m8.547745758s) [seg_pos={0 0}, upload=470746]: Done begin upload segment. Dec 28 00:15:08.676701146 (536h46m8.547784221s) [noise=true , address=77.90.31.16:4736 , piece_id=4ESIJ3UZV6AU77DABUFVONFXNBMBPRJQGF6X4LSIBJ7OIXY7LU2Q, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.676720973 (536h46m8.547804018s) [noise=true , address=145.79.227.52:4445 , piece_id=OLCOGVGMYPBQ3LDY2IS3X57UDYATY6FK5NU2JXGHCWPZ376YHSVA, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.676887721 (536h46m8.547970756s) [noise=true , address=145.79.227.49:4337 , piece_id=AKIGSSXRBM6MJQA7D2H6SKKT76DOUZSC5IPTYFSG6KARAW7BTWZQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.677303463 (536h46m8.548386508s) [noise=true , address=145.79.227.15:4873 , piece_id=2IVDPBLBJG7HWZ3I7XF64PL6CRRSWFWRUEO3BGPCYI3AILBW7SCQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.677314163 (536h46m8.548397198s) [noise=true , address=145.79.227.19:4078 , piece_id=GYOZXF6MXQX6CGPEJOV6KDSLOPUFQQ3ERP7V4XWMOKMW77CY73HA, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.677646366 (536h46m8.548729411s) [noise=true , address=145.79.227.31:4798 , piece_id=7Q2W25GRLDDQSLTXRBWRZ6JM7CKTXUEHJE4VTVRR33GREAXHQV3A, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.677652608 (536h46m8.548735653s) [noise=true , address=145.79.227.32:4836 , piece_id=V3BVXMBTCW3IA5EUBMAXNRKJ62KSTEHCZVXGCV5HQTNQ2FMQRWLQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.677972827 (536h46m8.549055862s) [noise=true , address=45.91.116.4:4940 , piece_id=2YIJZXP2EDO3ZNSLG7BXMKKOWLSLQBQ7IZIROQYMHAJM43WGR2SA, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.678032150 (536h46m8.549115185s) [noise=true , address=77.90.31.12:4590 , piece_id=REYB2K655GSM763ZUN4VS7YH3S37T6JNRZJ4XWK6YZDEEB4NEW6Q, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.678660537 (536h46m8.549743612s) [noise=true , address=45.91.116.9:4261 , piece_id=TGY3V36OS4V7SLFNCVU655EJHTV4INGJHWBYXE6TFCICP2TTLLKQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.679059757 (536h46m8.550142802s) [noise=true , address=145.79.227.37:4566 , piece_id=2MBITVMRJPF5OQESRNLV6DI2TEXF5DY7CAFQRKEX6SQHOVCWSMEA, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.679379977 (536h46m8.550463022s) [noise=true , address=145.79.227.41:4086 , piece_id=HAZ5Q5NIIYTSLU2YG6BWOPMIQ2REYGHTFHJMVMIXUJFKCDRC42KA, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.679587512 (536h46m8.550670557s) [noise=true , address=145.79.227.42:4122 , piece_id=EGOB7OL6SHQZXE7DYJQYZBME6QOFKPUKI6Y3YLKNMJCKBI3447PQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.679744451 (536h46m8.550827486s) [noise=true , address=145.79.227.51:4409 , piece_id=VUBKBPPODYQF2FS2NS3JQDZSIS5EAVBDG3BSIBXOEOS473KB4WTQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.680028902 (536h46m8.551111947s) [noise=true , address=145.79.227.46:4603 , piece_id=YH7VMT63SCTAUET4ZL434DHXZAD4VJWIK26KFQRFMLIWUDVIH6TQ, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:08.680701944 (536h46m8.551784979s) [noise=true , address=145.79.227.14:4839 , piece_id=RDO3EXV4O6XMBJCCBDYKBNBSH5JPKMQX24HYVP6OZR75F4KAHO2A, seg_pos={0 0}, upload=470746]: Uploading piece... Dec 28 00:15:10.383427059 (536h46m10.254510094s) [upload=470746]: Next returned nil segment Dec 28 00:15:10.383446646 (536h46m10.254529691s) [upload=470746]: Waiting for error group managing segments... Dec 28 00:15:10.670104790 (536h46m10.541187835s) [noise=true , address=145.79.227.32:4836 , piece_id=V3BVXMBTCW3IA5EUBMAXNRKJ62KSTEHCZVXGCV5HQTNQ2FMQRWLQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.678994417 (536h46m10.550077452s) [noise=true , address=145.79.227.52:4445 , piece_id=OLCOGVGMYPBQ3LDY2IS3X57UDYATY6FK5NU2JXGHCWPZ376YHSVA, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.679846740 (536h46m10.550929785s) [noise=true , address=145.79.227.46:4603 , piece_id=YH7VMT63SCTAUET4ZL434DHXZAD4VJWIK26KFQRFMLIWUDVIH6TQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.683718673 (536h46m10.554801718s) [noise=true , address=145.79.227.37:4566 , piece_id=2MBITVMRJPF5OQESRNLV6DI2TEXF5DY7CAFQRKEX6SQHOVCWSMEA, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.684543243 (536h46m10.555626278s) [noise=true , address=145.79.227.15:4873 , piece_id=2IVDPBLBJG7HWZ3I7XF64PL6CRRSWFWRUEO3BGPCYI3AILBW7SCQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.685269356 (536h46m10.556352401s) [noise=true , address=145.79.227.42:4122 , piece_id=EGOB7OL6SHQZXE7DYJQYZBME6QOFKPUKI6Y3YLKNMJCKBI3447PQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.690006967 (536h46m10.561090012s) [noise=true , address=145.79.227.31:4798 , piece_id=7Q2W25GRLDDQSLTXRBWRZ6JM7CKTXUEHJE4VTVRR33GREAXHQV3A, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.697989727 (536h46m10.569072773s) [noise=true , address=145.79.227.19:4078 , piece_id=GYOZXF6MXQX6CGPEJOV6KDSLOPUFQQ3ERP7V4XWMOKMW77CY73HA, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.699565818 (536h46m10.570648863s) [noise=true , address=145.79.227.51:4409 , piece_id=VUBKBPPODYQF2FS2NS3JQDZSIS5EAVBDG3BSIBXOEOS473KB4WTQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.718776609 (536h46m10.589859654s) [noise=true , address=145.79.227.14:4839 , piece_id=RDO3EXV4O6XMBJCCBDYKBNBSH5JPKMQX24HYVP6OZR75F4KAHO2A, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.908261129 (536h46m10.779344174s) [noise=true , address=77.90.31.12:4590 , piece_id=REYB2K655GSM763ZUN4VS7YH3S37T6JNRZJ4XWK6YZDEEB4NEW6Q, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:10.986794920 (536h46m10.857877955s) [noise=true , address=77.90.31.16:4736 , piece_id=4ESIJ3UZV6AU77DABUFVONFXNBMBPRJQGF6X4LSIBJ7OIXY7LU2Q, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:11.001995684 (536h46m10.873078729s) [noise=true , address=45.91.116.9:4261 , piece_id=TGY3V36OS4V7SLFNCVU655EJHTV4INGJHWBYXE6TFCICP2TTLLKQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:11.002018358 (536h46m10.873101393s) [seg_pos={0 0}, upload=470746]: Segment reached optimal threshold of 13 pieces. Dec 28 00:15:11.002062251 (536h46m10.873145296s) [noise=true , address=45.91.116.4:4940 , piece_id=2YIJZXP2EDO3ZNSLG7BXMKKOWLSLQBQ7IZIROQYMHAJM43WGR2SA, seg_pos={0 0}, upload=470746]: Done uploading piece. err: <nil> Dec 28 00:15:11.002247834 (536h46m10.873330879s) [noise=true , address=145.79.227.49:4337 , piece_id=AKIGSSXRBM6MJQA7D2H6SKKT76DOUZSC5IPTYFSG6KARAW7BTWZQ, seg_pos={0 0}, upload=470746]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 28 00:15:11.002256061 (536h46m10.873339106s) [noise=true , address=145.79.227.49:4337 , piece_id=AKIGSSXRBM6MJQA7D2H6SKKT76DOUZSC5IPTYFSG6KARAW7BTWZQ, seg_pos={0 0}, upload=470746]: Long tail cancellation due to optimal threshold Dec 28 00:15:11.002536735 (536h46m10.87361978s) [noise=true , address=145.79.227.41:4086 , piece_id=HAZ5Q5NIIYTSLU2YG6BWOPMIQ2REYGHTFHJMVMIXUJFKCDRC42KA, seg_pos={0 0}, upload=470746]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:34508->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:34508->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34508->145.79.227.41:4086: use of closed network connection Dec 28 00:15:11.002548447 (536h46m10.873631492s) [noise=true , address=145.79.227.41:4086 , piece_id=HAZ5Q5NIIYTSLU2YG6BWOPMIQ2REYGHTFHJMVMIXUJFKCDRC42KA, seg_pos={0 0}, upload=470746]: Long tail cancellation due to optimal threshold Dec 28 00:15:11.002573375 (536h46m10.87365641s) [seg_pos={0 0}, upload=470746]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 00:15:11.002678515 (536h46m10.87376155s) [upload=470746]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:15:11.002687191 (536h46m10.873770236s) [upload=470746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:15:11.289509848 (536h46m11.160592904s) [upload=470746]: Done uploading segments... Dec 28 00:15:11.289541920 (536h46m11.160624965s) [upload=470746]: Upload finished. err: <nil> Duration: 30.002391087s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.545940787s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:15:33 UTC) orbiter warning: context deadline exceeded Duration: 3.545955505s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.483353152s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:15:05 UTC) orbiter warning: context deadline exceeded Duration: 3.483405082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX.] duration: 20.604107552s trace: https://jaeger.dev.storj.io/trace/51583881dfa685b4 FAIL (2025-12-28 00:14:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: IEFXGO33WXELSMRL2NCYBX6XO7I7JX3RWO5XFKTCK4NCE4HF7V6Q) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 4ATMQHOC2X3U2BKOOW7AVPAPOIM3LF3ZQ23ECNUXV2L2XCNLQHNQ) context deadline exceeded ======LOGS====== Dec 28 00:13:29.098924145 (536h44m28.97000719s) [upload=470726]: Starting upload Dec 28 00:13:29.099013405 (536h44m28.97009645s) [upload=470726]: Done starting upload Dec 28 00:13:29.099120058 (536h44m28.970203134s) [upload=470726]: Uploading segments... Dec 28 00:13:29.099299580 (536h44m28.970382625s) [upload=470726]: Got next segment. Inline: false Dec 28 00:13:29.099393880 (536h44m28.970476945s) [upload=470726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:13:29.099400853 (536h44m28.970483908s) [upload=470726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:13:29.345748968 (536h44m29.216832013s) [seg_pos={0 0}, upload=470726]: Begin upload segment... Dec 28 00:13:29.345849639 (536h44m29.216932684s) [noise=true , address=145.79.227.38:4978 , piece_id=YWSDSBTXH3ZE76EG6AIQYB2YAPPPIEDR7WQJO5GUJC7KSZCIFKPA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.345862173 (536h44m29.216945208s) [seg_pos={0 0}, upload=470726]: Done begin upload segment. Dec 28 00:13:29.345891729 (536h44m29.216974764s) [noise=true , address=145.79.227.47:4267 , piece_id=OHBZGHGE3XYPAUTPPBJGPNU6NND7ZLR2E2NUE63CCSXBJHWIE6DA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.346557858 (536h44m29.217640903s) [noise=true , address=45.91.116.14:4192 , piece_id=SIQ33BHXXURRKJZTYW4XMNOBWDOPF6ENPLOTAQQLCY7ETERRYNIQ, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.346968339 (536h44m29.218051384s) [noise=true , address=145.79.227.46:4601 , piece_id=L72IYZSE7IZJ2LGXRNQNRXTWRKENALXBHGGBABW3CHSYOHY43VNA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.348246914 (536h44m29.219329949s) [noise=true , address=145.79.227.39:4015 , piece_id=6ABN6AQHHFENQ3MOZL22NNXUVANB6HCAUAKPFOTCPBYSG2J6G5JA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.348724182 (536h44m29.219807227s) [noise=true , address=77.90.31.17:4770 , piece_id=IEFXGO33WXELSMRL2NCYBX6XO7I7JX3RWO5XFKTCK4NCE4HF7V6Q, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.349722083 (536h44m29.220805118s) [noise=true , address=145.79.227.15:4874 , piece_id=XWC2P2CWB2G7QR7DETVX4HR53EXT5LEGP4QOSKSFFZ4GDSBNCCBA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.350153995 (536h44m29.22123704s) [noise=true , address=145.79.227.36:4518 , piece_id=YABDG3LKJ2XYYS45JCMAWFJTMBPNWVA2QU2ATLZPMSL72G7MUCJA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.350885468 (536h44m29.221968503s) [noise=true , address=145.79.227.6:4257 , piece_id=LPNLMLOEZGNUCX7YHNORCJONLTWH3TE4Y4IXY52C57HCLX3LCTDQ, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.351584479 (536h44m29.222667524s) [noise=true , address=145.79.227.48:4301 , piece_id=45HGXA7Q7UTWCECTNOS3V6R7KJHSTKVCJ3BA7V4XRKVV5HND6YEA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.355643989 (536h44m29.226727034s) [noise=true , address=145.79.227.12:4767 , piece_id=42QZI5ZT3QFLPNLBNI5S5VDXLR5VRHFBMCXODEWZUJQEQVRT7CNA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.357726425 (536h44m29.22880947s) [noise=true , address=77.90.31.14:4663 , piece_id=4ATMQHOC2X3U2BKOOW7AVPAPOIM3LF3ZQ23ECNUXV2L2XCNLQHNQ, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.359975768 (536h44m29.231058803s) [noise=true , address=45.91.116.13:4156 , piece_id=245QP24EVHHKXTKB5CLLSX6FQ5ZI4LK223WLCKVTUG3UIGJ3W7UQ, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.362699184 (536h44m29.233782239s) [noise=true , address=77.90.31.16:4735 , piece_id=537BOK34N3DDTOYK76GC5IUAQ2AUXDYCYZQHLMBBL2EMV3ICZ3JA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.363686724 (536h44m29.234769769s) [noise=true , address=145.79.227.21:4473 , piece_id=V52NNVHZN4BILKDOBVRP5PR6WGHLSBFEGN5XZIS4FSPMVENL45PQ, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.365282593 (536h44m29.236365638s) [noise=true , address=145.79.227.35:4482 , piece_id=IMGCQLVJIUBAUJK2R4A6ASRNB4GDZABXWFLP2YQPRDBLEO77DKPA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:31.064571130 (536h44m30.935654175s) [upload=470726]: Next returned nil segment Dec 28 00:13:31.064582922 (536h44m30.935665957s) [upload=470726]: Waiting for error group managing segments... Dec 28 00:13:31.349610178 (536h44m31.220693213s) [noise=true , address=145.79.227.46:4601 , piece_id=L72IYZSE7IZJ2LGXRNQNRXTWRKENALXBHGGBABW3CHSYOHY43VNA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.355041270 (536h44m31.226124305s) [noise=true , address=145.79.227.38:4978 , piece_id=YWSDSBTXH3ZE76EG6AIQYB2YAPPPIEDR7WQJO5GUJC7KSZCIFKPA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.372489764 (536h44m31.243572819s) [noise=true , address=145.79.227.36:4518 , piece_id=YABDG3LKJ2XYYS45JCMAWFJTMBPNWVA2QU2ATLZPMSL72G7MUCJA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.372772432 (536h44m31.243855467s) [noise=true , address=145.79.227.15:4874 , piece_id=XWC2P2CWB2G7QR7DETVX4HR53EXT5LEGP4QOSKSFFZ4GDSBNCCBA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.373791453 (536h44m31.244874488s) [noise=true , address=145.79.227.47:4267 , piece_id=OHBZGHGE3XYPAUTPPBJGPNU6NND7ZLR2E2NUE63CCSXBJHWIE6DA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.376284710 (536h44m31.247367756s) [noise=true , address=145.79.227.39:4015 , piece_id=6ABN6AQHHFENQ3MOZL22NNXUVANB6HCAUAKPFOTCPBYSG2J6G5JA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.391462691 (536h44m31.262545776s) [noise=true , address=145.79.227.12:4767 , piece_id=42QZI5ZT3QFLPNLBNI5S5VDXLR5VRHFBMCXODEWZUJQEQVRT7CNA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.393986096 (536h44m31.265069142s) [noise=true , address=145.79.227.48:4301 , piece_id=45HGXA7Q7UTWCECTNOS3V6R7KJHSTKVCJ3BA7V4XRKVV5HND6YEA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.394948570 (536h44m31.266031605s) [noise=true , address=145.79.227.21:4473 , piece_id=V52NNVHZN4BILKDOBVRP5PR6WGHLSBFEGN5XZIS4FSPMVENL45PQ, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.427642233 (536h44m31.298725278s) [noise=true , address=145.79.227.6:4257 , piece_id=LPNLMLOEZGNUCX7YHNORCJONLTWH3TE4Y4IXY52C57HCLX3LCTDQ, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.430422176 (536h44m31.301505211s) [noise=true , address=145.79.227.35:4482 , piece_id=IMGCQLVJIUBAUJK2R4A6ASRNB4GDZABXWFLP2YQPRDBLEO77DKPA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.549640476 (536h44m31.420723521s) [noise=true , address=77.90.31.14:4663 , piece_id=4ATMQHOC2X3U2BKOOW7AVPAPOIM3LF3ZQ23ECNUXV2L2XCNLQHNQ, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.610379984 (536h44m31.481463029s) [noise=true , address=77.90.31.17:4770 , piece_id=IEFXGO33WXELSMRL2NCYBX6XO7I7JX3RWO5XFKTCK4NCE4HF7V6Q, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.610408809 (536h44m31.481491854s) [seg_pos={0 0}, upload=470726]: Segment reached optimal threshold of 13 pieces. Dec 28 00:13:31.610615602 (536h44m31.481698637s) [noise=true , address=77.90.31.16:4735 , piece_id=537BOK34N3DDTOYK76GC5IUAQ2AUXDYCYZQHLMBBL2EMV3ICZ3JA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 00:13:31.610625972 (536h44m31.481709007s) [noise=true , address=77.90.31.16:4735 , piece_id=537BOK34N3DDTOYK76GC5IUAQ2AUXDYCYZQHLMBBL2EMV3ICZ3JA, seg_pos={0 0}, upload=470726]: Long tail cancellation due to optimal threshold Dec 28 00:13:31.610769376 (536h44m31.481852411s) [noise=true , address=45.91.116.13:4156 , piece_id=245QP24EVHHKXTKB5CLLSX6FQ5ZI4LK223WLCKVTUG3UIGJ3W7UQ, seg_pos={0 0}, upload=470726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 00:13:31.610786247 (536h44m31.481869293s) [noise=true , address=45.91.116.13:4156 , piece_id=245QP24EVHHKXTKB5CLLSX6FQ5ZI4LK223WLCKVTUG3UIGJ3W7UQ, seg_pos={0 0}, upload=470726]: Long tail cancellation due to optimal threshold Dec 28 00:13:31.610919331 (536h44m31.482002376s) [noise=true , address=45.91.116.14:4192 , piece_id=SIQ33BHXXURRKJZTYW4XMNOBWDOPF6ENPLOTAQQLCY7ETERRYNIQ, seg_pos={0 0}, upload=470726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 28 00:13:31.610927166 (536h44m31.482010211s) [noise=true , address=45.91.116.14:4192 , piece_id=SIQ33BHXXURRKJZTYW4XMNOBWDOPF6ENPLOTAQQLCY7ETERRYNIQ, seg_pos={0 0}, upload=470726]: Long tail cancellation due to optimal threshold Dec 28 00:13:31.610946643 (536h44m31.482029688s) [seg_pos={0 0}, upload=470726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:13:31.611014523 (536h44m31.482097558s) [upload=470726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:13:31.611021225 (536h44m31.48210427s) [upload=470726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:13:31.899343214 (536h44m31.770426259s) [upload=470726]: Done uploading segments... Dec 28 00:13:31.899364606 (536h44m31.770447641s) [upload=470726]: Upload finished. err: <nil> Duration: 20.604123803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XXX] duration: 30.001970939s trace: https://jaeger.dev.storj.io/trace/51583881dfa685b4 FAIL (2025-12-28 00:13:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: IEFXGO33WXELSMRL2NCYBX6XO7I7JX3RWO5XFKTCK4NCE4HF7V6Q) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 4ATMQHOC2X3U2BKOOW7AVPAPOIM3LF3ZQ23ECNUXV2L2XCNLQHNQ) context deadline exceeded ======LOGS====== Dec 28 00:13:29.098924145 (536h44m28.97000719s) [upload=470726]: Starting upload Dec 28 00:13:29.099013405 (536h44m28.97009645s) [upload=470726]: Done starting upload Dec 28 00:13:29.099120058 (536h44m28.970203134s) [upload=470726]: Uploading segments... Dec 28 00:13:29.099299580 (536h44m28.970382625s) [upload=470726]: Got next segment. Inline: false Dec 28 00:13:29.099393880 (536h44m28.970476945s) [upload=470726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:13:29.099400853 (536h44m28.970483908s) [upload=470726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:13:29.345748968 (536h44m29.216832013s) [seg_pos={0 0}, upload=470726]: Begin upload segment... Dec 28 00:13:29.345849639 (536h44m29.216932684s) [noise=true , address=145.79.227.38:4978 , piece_id=YWSDSBTXH3ZE76EG6AIQYB2YAPPPIEDR7WQJO5GUJC7KSZCIFKPA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.345862173 (536h44m29.216945208s) [seg_pos={0 0}, upload=470726]: Done begin upload segment. Dec 28 00:13:29.345891729 (536h44m29.216974764s) [noise=true , address=145.79.227.47:4267 , piece_id=OHBZGHGE3XYPAUTPPBJGPNU6NND7ZLR2E2NUE63CCSXBJHWIE6DA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.346557858 (536h44m29.217640903s) [noise=true , address=45.91.116.14:4192 , piece_id=SIQ33BHXXURRKJZTYW4XMNOBWDOPF6ENPLOTAQQLCY7ETERRYNIQ, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.346968339 (536h44m29.218051384s) [noise=true , address=145.79.227.46:4601 , piece_id=L72IYZSE7IZJ2LGXRNQNRXTWRKENALXBHGGBABW3CHSYOHY43VNA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.348246914 (536h44m29.219329949s) [noise=true , address=145.79.227.39:4015 , piece_id=6ABN6AQHHFENQ3MOZL22NNXUVANB6HCAUAKPFOTCPBYSG2J6G5JA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.348724182 (536h44m29.219807227s) [noise=true , address=77.90.31.17:4770 , piece_id=IEFXGO33WXELSMRL2NCYBX6XO7I7JX3RWO5XFKTCK4NCE4HF7V6Q, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.349722083 (536h44m29.220805118s) [noise=true , address=145.79.227.15:4874 , piece_id=XWC2P2CWB2G7QR7DETVX4HR53EXT5LEGP4QOSKSFFZ4GDSBNCCBA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.350153995 (536h44m29.22123704s) [noise=true , address=145.79.227.36:4518 , piece_id=YABDG3LKJ2XYYS45JCMAWFJTMBPNWVA2QU2ATLZPMSL72G7MUCJA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.350885468 (536h44m29.221968503s) [noise=true , address=145.79.227.6:4257 , piece_id=LPNLMLOEZGNUCX7YHNORCJONLTWH3TE4Y4IXY52C57HCLX3LCTDQ, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.351584479 (536h44m29.222667524s) [noise=true , address=145.79.227.48:4301 , piece_id=45HGXA7Q7UTWCECTNOS3V6R7KJHSTKVCJ3BA7V4XRKVV5HND6YEA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.355643989 (536h44m29.226727034s) [noise=true , address=145.79.227.12:4767 , piece_id=42QZI5ZT3QFLPNLBNI5S5VDXLR5VRHFBMCXODEWZUJQEQVRT7CNA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.357726425 (536h44m29.22880947s) [noise=true , address=77.90.31.14:4663 , piece_id=4ATMQHOC2X3U2BKOOW7AVPAPOIM3LF3ZQ23ECNUXV2L2XCNLQHNQ, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.359975768 (536h44m29.231058803s) [noise=true , address=45.91.116.13:4156 , piece_id=245QP24EVHHKXTKB5CLLSX6FQ5ZI4LK223WLCKVTUG3UIGJ3W7UQ, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.362699184 (536h44m29.233782239s) [noise=true , address=77.90.31.16:4735 , piece_id=537BOK34N3DDTOYK76GC5IUAQ2AUXDYCYZQHLMBBL2EMV3ICZ3JA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.363686724 (536h44m29.234769769s) [noise=true , address=145.79.227.21:4473 , piece_id=V52NNVHZN4BILKDOBVRP5PR6WGHLSBFEGN5XZIS4FSPMVENL45PQ, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:29.365282593 (536h44m29.236365638s) [noise=true , address=145.79.227.35:4482 , piece_id=IMGCQLVJIUBAUJK2R4A6ASRNB4GDZABXWFLP2YQPRDBLEO77DKPA, seg_pos={0 0}, upload=470726]: Uploading piece... Dec 28 00:13:31.064571130 (536h44m30.935654175s) [upload=470726]: Next returned nil segment Dec 28 00:13:31.064582922 (536h44m30.935665957s) [upload=470726]: Waiting for error group managing segments... Dec 28 00:13:31.349610178 (536h44m31.220693213s) [noise=true , address=145.79.227.46:4601 , piece_id=L72IYZSE7IZJ2LGXRNQNRXTWRKENALXBHGGBABW3CHSYOHY43VNA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.355041270 (536h44m31.226124305s) [noise=true , address=145.79.227.38:4978 , piece_id=YWSDSBTXH3ZE76EG6AIQYB2YAPPPIEDR7WQJO5GUJC7KSZCIFKPA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.372489764 (536h44m31.243572819s) [noise=true , address=145.79.227.36:4518 , piece_id=YABDG3LKJ2XYYS45JCMAWFJTMBPNWVA2QU2ATLZPMSL72G7MUCJA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.372772432 (536h44m31.243855467s) [noise=true , address=145.79.227.15:4874 , piece_id=XWC2P2CWB2G7QR7DETVX4HR53EXT5LEGP4QOSKSFFZ4GDSBNCCBA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.373791453 (536h44m31.244874488s) [noise=true , address=145.79.227.47:4267 , piece_id=OHBZGHGE3XYPAUTPPBJGPNU6NND7ZLR2E2NUE63CCSXBJHWIE6DA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.376284710 (536h44m31.247367756s) [noise=true , address=145.79.227.39:4015 , piece_id=6ABN6AQHHFENQ3MOZL22NNXUVANB6HCAUAKPFOTCPBYSG2J6G5JA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.391462691 (536h44m31.262545776s) [noise=true , address=145.79.227.12:4767 , piece_id=42QZI5ZT3QFLPNLBNI5S5VDXLR5VRHFBMCXODEWZUJQEQVRT7CNA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.393986096 (536h44m31.265069142s) [noise=true , address=145.79.227.48:4301 , piece_id=45HGXA7Q7UTWCECTNOS3V6R7KJHSTKVCJ3BA7V4XRKVV5HND6YEA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.394948570 (536h44m31.266031605s) [noise=true , address=145.79.227.21:4473 , piece_id=V52NNVHZN4BILKDOBVRP5PR6WGHLSBFEGN5XZIS4FSPMVENL45PQ, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.427642233 (536h44m31.298725278s) [noise=true , address=145.79.227.6:4257 , piece_id=LPNLMLOEZGNUCX7YHNORCJONLTWH3TE4Y4IXY52C57HCLX3LCTDQ, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.430422176 (536h44m31.301505211s) [noise=true , address=145.79.227.35:4482 , piece_id=IMGCQLVJIUBAUJK2R4A6ASRNB4GDZABXWFLP2YQPRDBLEO77DKPA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.549640476 (536h44m31.420723521s) [noise=true , address=77.90.31.14:4663 , piece_id=4ATMQHOC2X3U2BKOOW7AVPAPOIM3LF3ZQ23ECNUXV2L2XCNLQHNQ, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.610379984 (536h44m31.481463029s) [noise=true , address=77.90.31.17:4770 , piece_id=IEFXGO33WXELSMRL2NCYBX6XO7I7JX3RWO5XFKTCK4NCE4HF7V6Q, seg_pos={0 0}, upload=470726]: Done uploading piece. err: <nil> Dec 28 00:13:31.610408809 (536h44m31.481491854s) [seg_pos={0 0}, upload=470726]: Segment reached optimal threshold of 13 pieces. Dec 28 00:13:31.610615602 (536h44m31.481698637s) [noise=true , address=77.90.31.16:4735 , piece_id=537BOK34N3DDTOYK76GC5IUAQ2AUXDYCYZQHLMBBL2EMV3ICZ3JA, seg_pos={0 0}, upload=470726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 00:13:31.610625972 (536h44m31.481709007s) [noise=true , address=77.90.31.16:4735 , piece_id=537BOK34N3DDTOYK76GC5IUAQ2AUXDYCYZQHLMBBL2EMV3ICZ3JA, seg_pos={0 0}, upload=470726]: Long tail cancellation due to optimal threshold Dec 28 00:13:31.610769376 (536h44m31.481852411s) [noise=true , address=45.91.116.13:4156 , piece_id=245QP24EVHHKXTKB5CLLSX6FQ5ZI4LK223WLCKVTUG3UIGJ3W7UQ, seg_pos={0 0}, upload=470726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 00:13:31.610786247 (536h44m31.481869293s) [noise=true , address=45.91.116.13:4156 , piece_id=245QP24EVHHKXTKB5CLLSX6FQ5ZI4LK223WLCKVTUG3UIGJ3W7UQ, seg_pos={0 0}, upload=470726]: Long tail cancellation due to optimal threshold Dec 28 00:13:31.610919331 (536h44m31.482002376s) [noise=true , address=45.91.116.14:4192 , piece_id=SIQ33BHXXURRKJZTYW4XMNOBWDOPF6ENPLOTAQQLCY7ETERRYNIQ, seg_pos={0 0}, upload=470726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 28 00:13:31.610927166 (536h44m31.482010211s) [noise=true , address=45.91.116.14:4192 , piece_id=SIQ33BHXXURRKJZTYW4XMNOBWDOPF6ENPLOTAQQLCY7ETERRYNIQ, seg_pos={0 0}, upload=470726]: Long tail cancellation due to optimal threshold Dec 28 00:13:31.610946643 (536h44m31.482029688s) [seg_pos={0 0}, upload=470726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:13:31.611014523 (536h44m31.482097558s) [upload=470726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:13:31.611021225 (536h44m31.48210427s) [upload=470726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:13:31.899343214 (536h44m31.770426259s) [upload=470726]: Done uploading segments... Dec 28 00:13:31.899364606 (536h44m31.770447641s) [upload=470726]: Upload finished. err: <nil> Duration: 30.001994125s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.565712488s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:13:51 UTC) orbiter warning: context deadline exceeded Duration: 3.565754528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.532629504s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:13:09 UTC) orbiter warning: context deadline exceeded Duration: 3.532657776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.XX] duration: 30.005349s trace: https://jaeger.dev.storj.io/trace/1e15613b00491c59 FAIL (2025-12-28 00:13:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: FSEF4LRWYAHJWBKRCYWZCGSGYARG64VDWN2R5PMRRBJKPWDUSMIQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: I2WJKI7ZC34L7LW7QEHU6K4GGXNJWGKJH2BNU5YAOLVRGZD2R7RA) context deadline exceeded ======LOGS====== 0:12:31.092488532 (536h43m30.963571567s) [upload=470710]: Starting upload Dec 28 00:12:31.092604883 (536h43m30.963687918s) [upload=470710]: Done starting upload Dec 28 00:12:31.092664116 (536h43m30.963747161s) [upload=470710]: Uploading segments... Dec 28 00:12:31.092836885 (536h43m30.96391993s) [upload=470710]: Got next segment. Inline: false Dec 28 00:12:31.092881540 (536h43m30.963964575s) [upload=470710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:12:31.092887942 (536h43m30.963970977s) [upload=470710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:12:31.340050780 (536h43m31.211133825s) [seg_pos={0 0}, upload=470710]: Begin upload segment... Dec 28 00:12:31.340117487 (536h43m31.211200532s) [seg_pos={0 0}, upload=470710]: Done begin upload segment. Dec 28 00:12:31.340134469 (536h43m31.211217514s) [noise=true , address=145.79.227.19:4077 , piece_id=DT73N2ZNU5ESUM6U5BJ3W7ZAKFJVEOJU74SK44ZAP4DU4P7GWE6A, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.340196407 (536h43m31.211279452s) [noise=true , address=145.79.227.18:4006 , piece_id=HZFRCEJQT5XUQM4EGFYD7XUZLFGIBINZYJJA6KY23EZDDZRIBOYQ, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.340584085 (536h43m31.21166712s) [noise=true , address=145.79.227.15:4874 , piece_id=PMQ6FTYF7T6GDCDOFWEDPLOJZDLO34BWJM7NM7QVR6RAIIJJVDQA, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.341336458 (536h43m31.212419503s) [noise=true , address=77.90.31.11:4556 , piece_id=I2WJKI7ZC34L7LW7QEHU6K4GGXNJWGKJH2BNU5YAOLVRGZD2R7RA, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.341643572 (536h43m31.212726617s) [noise=true , address=77.90.31.13:4626 , piece_id=P7UETMR3YL4VXBBN7K7DIVJ4PKLSEQMOTQTIRYMK7WJOWD4RSC2A, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.342054274 (536h43m31.213137319s) [noise=true , address=145.79.227.9:4367 , piece_id=5NPPRDA53G5VSSSW45BVABFKMIE7RNPSU5RQNI4UWEHJLRB7XB3Q, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.342540941 (536h43m31.213623986s) [noise=true , address=145.79.227.40:4050 , piece_id=WOCCEK64F3LGYAYUERXEO5MMBANGHM7LMFB427KK35FIN2INQSVQ, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.342970188 (536h43m31.214053233s) [noise=true , address=145.79.227.29:4942 , piece_id=3HLC4UWITEKMOOWJOEYDAUSGMWDQAYGL7BBAW6GAEW746ENVFKCQ, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.344188878 (536h43m31.215271913s) [noise=true , address=145.79.227.36:4518 , piece_id=ASBZQG7D6TXDRONFZX3XRQ4DMA4RJATOPA7AAGBFZVVJ4V5MOJHQ, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.344378240 (536h43m31.215461285s) [noise=true , address=145.79.227.52:4446 , piece_id=JVXZTUFGBQMQT64KA6GLJ5X4AGYT7CGQ7M2GMXXR4AZU7B62QN3Q, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.345629783 (536h43m31.216712818s) [noise=true , address=45.91.116.4:4939 , piece_id=U6AXNVI4WNDEVY32ZM77KSKQOYVCOTJ74VVP5WFZ7ZVHNHNF3WBA, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.345681491 (536h43m31.216764536s) [noise=true , address=45.91.116.10:4082 , piece_id=XBWICUVGYYXBWDYUAJKY6KPGBVJ2WM7JWHRKEONXL7ERFB7QZRCA, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.346932132 (536h43m31.218015167s) [noise=true , address=45.91.116.5:4299 , piece_id=FSEF4LRWYAHJWBKRCYWZCGSGYARG64VDWN2R5PMRRBJKPWDUSMIQ, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.348465732 (536h43m31.219548767s) [noise=true , address=145.79.227.12:4767 , piece_id=ALW4ABIPFZFYS7VTE4JTE6HVKHVAEENRVUX2XO52UPRMJTZNJPBQ, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.349533646 (536h43m31.220616691s) [noise=true , address=145.79.227.13:4804 , piece_id=VEZX5DGF65DI55ZS5SJMBOEUIN4CVVE46STKC73DIMDJIEQ5NFWQ, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:31.352633619 (536h43m31.223716654s) [noise=true , address=45.91.116.13:4153 , piece_id=HQTGCB4U5JEOY74PJFSBZS7CI7Z7GIYKP6423J4VC3EDUSWXQTNQ, seg_pos={0 0}, upload=470710]: Uploading piece... Dec 28 00:12:33.071025462 (536h43m32.942108497s) [upload=470710]: Next returned nil segment Dec 28 00:12:33.071038768 (536h43m32.942121803s) [upload=470710]: Waiting for error group managing segments... Dec 28 00:12:33.356310630 (536h43m33.227393675s) [noise=true , address=145.79.227.15:4874 , piece_id=PMQ6FTYF7T6GDCDOFWEDPLOJZDLO34BWJM7NM7QVR6RAIIJJVDQA, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.356345917 (536h43m33.227428952s) [noise=true , address=145.79.227.13:4804 , piece_id=VEZX5DGF65DI55ZS5SJMBOEUIN4CVVE46STKC73DIMDJIEQ5NFWQ, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.361910492 (536h43m33.232993527s) [noise=true , address=145.79.227.52:4446 , piece_id=JVXZTUFGBQMQT64KA6GLJ5X4AGYT7CGQ7M2GMXXR4AZU7B62QN3Q, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.371019468 (536h43m33.242102503s) [noise=true , address=145.79.227.9:4367 , piece_id=5NPPRDA53G5VSSSW45BVABFKMIE7RNPSU5RQNI4UWEHJLRB7XB3Q, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.372501430 (536h43m33.243584475s) [noise=true , address=145.79.227.36:4518 , piece_id=ASBZQG7D6TXDRONFZX3XRQ4DMA4RJATOPA7AAGBFZVVJ4V5MOJHQ, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.372843651 (536h43m33.243926686s) [noise=true , address=145.79.227.19:4077 , piece_id=DT73N2ZNU5ESUM6U5BJ3W7ZAKFJVEOJU74SK44ZAP4DU4P7GWE6A, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.576850782 (536h43m33.447933827s) [noise=true , address=45.91.116.10:4082 , piece_id=XBWICUVGYYXBWDYUAJKY6KPGBVJ2WM7JWHRKEONXL7ERFB7QZRCA, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.594265011 (536h43m33.465348047s) [noise=true , address=145.79.227.12:4767 , piece_id=ALW4ABIPFZFYS7VTE4JTE6HVKHVAEENRVUX2XO52UPRMJTZNJPBQ, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.603352215 (536h43m33.47443526s) [noise=true , address=45.91.116.13:4153 , piece_id=HQTGCB4U5JEOY74PJFSBZS7CI7Z7GIYKP6423J4VC3EDUSWXQTNQ, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.606719457 (536h43m33.477802502s) [noise=true , address=77.90.31.13:4626 , piece_id=P7UETMR3YL4VXBBN7K7DIVJ4PKLSEQMOTQTIRYMK7WJOWD4RSC2A, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.615978578 (536h43m33.487061623s) [noise=true , address=45.91.116.5:4299 , piece_id=FSEF4LRWYAHJWBKRCYWZCGSGYARG64VDWN2R5PMRRBJKPWDUSMIQ, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.684569065 (536h43m33.55565211s) [noise=true , address=45.91.116.4:4939 , piece_id=U6AXNVI4WNDEVY32ZM77KSKQOYVCOTJ74VVP5WFZ7ZVHNHNF3WBA, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.829753347 (536h43m33.700836392s) [noise=true , address=77.90.31.11:4556 , piece_id=I2WJKI7ZC34L7LW7QEHU6K4GGXNJWGKJH2BNU5YAOLVRGZD2R7RA, seg_pos={0 0}, upload=470710]: Done uploading piece. err: <nil> Dec 28 00:12:33.829779257 (536h43m33.700862302s) [seg_pos={0 0}, upload=470710]: Segment reached optimal threshold of 13 pieces. Dec 28 00:12:33.830382295 (536h43m33.70146535s) [noise=true , address=145.79.227.29:4942 , piece_id=3HLC4UWITEKMOOWJOEYDAUSGMWDQAYGL7BBAW6GAEW746ENVFKCQ, seg_pos={0 0}, upload=470710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 28 00:12:33.830391964 (536h43m33.701474999s) [noise=true , address=145.79.227.29:4942 , piece_id=3HLC4UWITEKMOOWJOEYDAUSGMWDQAYGL7BBAW6GAEW746ENVFKCQ, seg_pos={0 0}, upload=470710]: Long tail cancellation due to optimal threshold Dec 28 00:12:33.830638784 (536h43m33.701721829s) [noise=true , address=145.79.227.40:4050 , piece_id=WOCCEK64F3LGYAYUERXEO5MMBANGHM7LMFB427KK35FIN2INQSVQ, seg_pos={0 0}, upload=470710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 00:12:33.830647129 (536h43m33.701730184s) [noise=true , address=145.79.227.40:4050 , piece_id=WOCCEK64F3LGYAYUERXEO5MMBANGHM7LMFB427KK35FIN2INQSVQ, seg_pos={0 0}, upload=470710]: Long tail cancellation due to optimal threshold Dec 28 00:12:33.830813306 (536h43m33.701896351s) [noise=true , address=145.79.227.18:4006 , piece_id=HZFRCEJQT5XUQM4EGFYD7XUZLFGIBINZYJJA6KY23EZDDZRIBOYQ, seg_pos={0 0}, upload=470710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:40600->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:40600->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40600->145.79.227.18:4006: use of closed network connection Dec 28 00:12:33.830833414 (536h43m33.701916469s) [noise=true , address=145.79.227.18:4006 , piece_id=HZFRCEJQT5XUQM4EGFYD7XUZLFGIBINZYJJA6KY23EZDDZRIBOYQ, seg_pos={0 0}, upload=470710]: Long tail cancellation due to optimal threshold Dec 28 00:12:33.830853202 (536h43m33.701936247s) [seg_pos={0 0}, upload=470710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:12:33.830932082 (536h43m33.702015127s) [upload=470710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:12:33.830938504 (536h43m33.702021549s) [upload=470710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:12:34.130273002 (536h43m34.001356037s) [upload=470710]: Done uploading segments... Dec 28 00:12:34.130339890 (536h43m34.001422925s) [upload=470710]: Upload finished. err: <nil> Duration: 30.005378095s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.X] duration: 30.00281547s trace: https://jaeger.dev.storj.io/trace/75bdc9be3b88ea85 FAIL (2025-12-28 00:12:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: H4BI5GIP32G7VX2VMYERFXKNUQ5MDZY2CVU2D734W2GL56H2HWZA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 2FFQTAL7DXXLDAXPONQZXMM2AJYMSEHDHBGTHII55T6VX3CALBQQ) context deadline exceeded ======LOGS====== 00:11:53.051165520 (536h42m52.922248565s) [upload=470698]: Starting upload Dec 28 00:11:53.051256342 (536h42m52.922339387s) [upload=470698]: Done starting upload Dec 28 00:11:53.051322017 (536h42m52.922405052s) [upload=470698]: Uploading segments... Dec 28 00:11:53.051516358 (536h42m52.922599403s) [upload=470698]: Got next segment. Inline: false Dec 28 00:11:53.051551475 (536h42m52.92263451s) [upload=470698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:11:53.051557305 (536h42m52.922640351s) [upload=470698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:11:53.296959769 (536h42m53.168042815s) [seg_pos={0 0}, upload=470698]: Begin upload segment... Dec 28 00:11:53.297002100 (536h42m53.168085135s) [seg_pos={0 0}, upload=470698]: Done begin upload segment. Dec 28 00:11:53.297025655 (536h42m53.16810869s) [noise=true , address=77.90.31.10:4517 , piece_id=HPO3JO4CNGWPARSHSPEOBLD34M3ECK52R2ZRI4U6TSTKGBVEIV7A, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.297102401 (536h42m53.168185446s) [noise=true , address=45.91.116.5:4297 , piece_id=JV2ECXON2WFC446METDNNKF4LZRGCIYHOS4C7QQ3YTHO6DYJEMGA, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.297683778 (536h42m53.168766813s) [noise=true , address=145.79.227.32:4836 , piece_id=DV7R4GSCF63Y6IGEUMFJMA64AEEK5EGN6RYUMKBSRSA3THL3ATDQ, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.297865815 (536h42m53.16894886s) [noise=true , address=145.79.227.13:4804 , piece_id=BV4LEUCLKKPM4DSPOTEX4LHNJ6JVCQEFJTYPYTG5C6CV5OJXPG3A, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.298069673 (536h42m53.169152708s) [noise=true , address=77.90.31.21:4878 , piece_id=5TZVFPAYRRESLPMNKJG6KEG4MUXBKJHYON6MGTYREQZXCKQIBO7A, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.298446150 (536h42m53.169529185s) [noise=true , address=45.91.116.14:4190 , piece_id=H4BI5GIP32G7VX2VMYERFXKNUQ5MDZY2CVU2D734W2GL56H2HWZA, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.298527144 (536h42m53.169610189s) [noise=true , address=45.91.116.9:4262 , piece_id=2FFQTAL7DXXLDAXPONQZXMM2AJYMSEHDHBGTHII55T6VX3CALBQQ, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.298830883 (536h42m53.169913918s) [noise=true , address=145.79.227.36:4519 , piece_id=XQC5VV5HIBHHZAF5NWCFH57IQNCMXMTXA6LNEVFI7ICKXSOE7KDA, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.298842244 (536h42m53.169925279s) [noise=true , address=77.90.31.14:4661 , piece_id=5JGKBT65BZKL4W4ZXXW3K6UNJLEICTBWZFGZI6HU2Y7EONE4BQYA, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.299164948 (536h42m53.170247983s) [noise=true , address=145.79.227.18:4007 , piece_id=5AQGSBL727V3ML7OQJZSLBY7DWFVZTO2I7MOHVKKDLMR3FIL3QZA, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.299187171 (536h42m53.170270216s) [noise=true , address=45.91.116.11:4117 , piece_id=JGVZ3YYC7PEVK6AVY26WGF5P2JPQSGVRDUDFGIDZR7VPN25OJAWQ, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.299519624 (536h42m53.170602669s) [noise=true , address=145.79.227.45:4231 , piece_id=VGVCX4Z6QSTTYL5VF5SXVQ5LHFXWNDREEHI6UMOAVRJSHE2KOSZQ, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.299656114 (536h42m53.170739149s) [noise=true , address=77.90.31.16:4735 , piece_id=GHP5CB53566744XBYMPKA5HIG3DMASDBJURZPCAC2JQGLOOA3PIQ, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.299881784 (536h42m53.170964819s) [noise=true , address=145.79.227.14:4838 , piece_id=MNTECNP4ITCWN2OSXV7CKEQ27B5N3M72XPLWO5WAXJKCE7OYLY4Q, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.300061776 (536h42m53.171144811s) [noise=true , address=145.79.227.52:4446 , piece_id=52PVBNFLVJZFMM2UNPRNXHBZHQUJGQDGH2N2CURESAHFC4L3WNHQ, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:53.300212764 (536h42m53.171295799s) [noise=true , address=145.79.227.6:4258 , piece_id=4QGOCD672ABXST7HOTQZAP76RAM2477IL7E63JKLVSRG2XKV5UXA, seg_pos={0 0}, upload=470698]: Uploading piece... Dec 28 00:11:55.036610654 (536h42m54.907693699s) [upload=470698]: Next returned nil segment Dec 28 00:11:55.036689965 (536h42m54.90777303s) [upload=470698]: Waiting for error group managing segments... Dec 28 00:11:55.318640963 (536h42m55.189723998s) [noise=true , address=145.79.227.14:4838 , piece_id=MNTECNP4ITCWN2OSXV7CKEQ27B5N3M72XPLWO5WAXJKCE7OYLY4Q, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.322511623 (536h42m55.193594658s) [noise=true , address=145.79.227.32:4836 , piece_id=DV7R4GSCF63Y6IGEUMFJMA64AEEK5EGN6RYUMKBSRSA3THL3ATDQ, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.324778239 (536h42m55.195861274s) [noise=true , address=145.79.227.13:4804 , piece_id=BV4LEUCLKKPM4DSPOTEX4LHNJ6JVCQEFJTYPYTG5C6CV5OJXPG3A, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.325397829 (536h42m55.196480864s) [noise=true , address=145.79.227.36:4519 , piece_id=XQC5VV5HIBHHZAF5NWCFH57IQNCMXMTXA6LNEVFI7ICKXSOE7KDA, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.328424382 (536h42m55.199507418s) [noise=true , address=145.79.227.45:4231 , piece_id=VGVCX4Z6QSTTYL5VF5SXVQ5LHFXWNDREEHI6UMOAVRJSHE2KOSZQ, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.334464012 (536h42m55.205547048s) [noise=true , address=145.79.227.18:4007 , piece_id=5AQGSBL727V3ML7OQJZSLBY7DWFVZTO2I7MOHVKKDLMR3FIL3QZA, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.358249334 (536h42m55.229332379s) [noise=true , address=145.79.227.52:4446 , piece_id=52PVBNFLVJZFMM2UNPRNXHBZHQUJGQDGH2N2CURESAHFC4L3WNHQ, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.519263149 (536h42m55.390346194s) [noise=true , address=45.91.116.9:4262 , piece_id=2FFQTAL7DXXLDAXPONQZXMM2AJYMSEHDHBGTHII55T6VX3CALBQQ, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.543340676 (536h42m55.414423721s) [noise=true , address=45.91.116.14:4190 , piece_id=H4BI5GIP32G7VX2VMYERFXKNUQ5MDZY2CVU2D734W2GL56H2HWZA, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.595265169 (536h42m55.466348214s) [noise=true , address=77.90.31.14:4661 , piece_id=5JGKBT65BZKL4W4ZXXW3K6UNJLEICTBWZFGZI6HU2Y7EONE4BQYA, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.607961234 (536h42m55.479044269s) [noise=true , address=45.91.116.5:4297 , piece_id=JV2ECXON2WFC446METDNNKF4LZRGCIYHOS4C7QQ3YTHO6DYJEMGA, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:55.624169306 (536h42m55.495252371s) [noise=true , address=77.90.31.21:4878 , piece_id=5TZVFPAYRRESLPMNKJG6KEG4MUXBKJHYON6MGTYREQZXCKQIBO7A, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:56.197117629 (536h42m56.068200664s) [noise=true , address=45.91.116.11:4117 , piece_id=JGVZ3YYC7PEVK6AVY26WGF5P2JPQSGVRDUDFGIDZR7VPN25OJAWQ, seg_pos={0 0}, upload=470698]: Done uploading piece. err: <nil> Dec 28 00:11:56.197137056 (536h42m56.068220091s) [seg_pos={0 0}, upload=470698]: Segment reached optimal threshold of 13 pieces. Dec 28 00:11:56.197294766 (536h42m56.068377801s) [noise=true , address=77.90.31.10:4517 , piece_id=HPO3JO4CNGWPARSHSPEOBLD34M3ECK52R2ZRI4U6TSTKGBVEIV7A, seg_pos={0 0}, upload=470698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 00:11:56.197300577 (536h42m56.068383612s) [noise=true , address=77.90.31.10:4517 , piece_id=HPO3JO4CNGWPARSHSPEOBLD34M3ECK52R2ZRI4U6TSTKGBVEIV7A, seg_pos={0 0}, upload=470698]: Long tail cancellation due to optimal threshold Dec 28 00:11:56.197353448 (536h42m56.068436483s) [noise=true , address=77.90.31.16:4735 , piece_id=GHP5CB53566744XBYMPKA5HIG3DMASDBJURZPCAC2JQGLOOA3PIQ, seg_pos={0 0}, upload=470698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 00:11:56.197362074 (536h42m56.068445119s) [noise=true , address=77.90.31.16:4735 , piece_id=GHP5CB53566744XBYMPKA5HIG3DMASDBJURZPCAC2JQGLOOA3PIQ, seg_pos={0 0}, upload=470698]: Long tail cancellation due to optimal threshold Dec 28 00:11:56.197512981 (536h42m56.068596016s) [noise=true , address=145.79.227.6:4258 , piece_id=4QGOCD672ABXST7HOTQZAP76RAM2477IL7E63JKLVSRG2XKV5UXA, seg_pos={0 0}, upload=470698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56522->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56522->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56522->145.79.227.6:4258: use of closed network connection Dec 28 00:11:56.197519314 (536h42m56.068602349s) [noise=true , address=145.79.227.6:4258 , piece_id=4QGOCD672ABXST7HOTQZAP76RAM2477IL7E63JKLVSRG2XKV5UXA, seg_pos={0 0}, upload=470698]: Long tail cancellation due to optimal threshold Dec 28 00:11:56.197537017 (536h42m56.068620052s) [seg_pos={0 0}, upload=470698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:11:56.197608844 (536h42m56.068691889s) [upload=470698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:11:56.197613302 (536h42m56.068696337s) [upload=470698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:11:56.490732046 (536h42m56.361815081s) [upload=470698]: Done uploading segments... Dec 28 00:11:56.490758737 (536h42m56.361841772s) [upload=470698]: Upload finished. err: <nil> Duration: 30.002838364s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.44467254s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:11:57 UTC) orbiter warning: context deadline exceeded Duration: 3.444695554s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.418212053s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:11:22 UTC) orbiter warning: context deadline exceeded Duration: 3.418257951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X.] duration: 6.755694317s trace: https://jaeger.dev.storj.io/trace/22b2b8c81bfa4fba FAIL (2025-12-28 00:11:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: QIAZKVEZIYNDDARO6SDGDRZAAPZT7NV4KRPM35YQ2HT7BWUUHQWQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: G6HEW6ESNVUAM2KZCZNRU3NBIO3QQJVC2JEYKQ636VQNXDQN5E4A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: FO7VXYZMLOD6HGJDDC7XW72NSPEYUKXHIP7UZTEL2T7BL22X4FGQ) context deadline exceeded ======LOGS====== ments... Dec 28 00:09:56.203334201 (536h40m56.074417246s) [upload=470674]: Got next segment. Inline: false Dec 28 00:09:56.203413443 (536h40m56.074496478s) [upload=470674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:09:56.203420416 (536h40m56.074503451s) [upload=470674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:09:56.450663154 (536h40m56.321746199s) [seg_pos={0 0}, upload=470674]: Begin upload segment... Dec 28 00:09:56.450704323 (536h40m56.321787358s) [seg_pos={0 0}, upload=470674]: Done begin upload segment. Dec 28 00:09:56.450736444 (536h40m56.321819489s) [noise=true , address=145.79.227.41:4085 , piece_id=RIVEHAZY7SVH5YIWV3AVTN4FEFYLWPQVR25GPTWCW3ZU6DQUQ5QA, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.450831355 (536h40m56.32191439s) [noise=true , address=145.79.227.38:4979 , piece_id=IJNTI3OXIBUXPKSI2SBZX34UDSJPHWW7EHJSK3O546XLCMJHMQUA, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.451450284 (536h40m56.322533339s) [noise=true , address=145.79.227.51:4411 , piece_id=Z6Y26CT7UFG3L3YFGVHNCU76ND6IIVCXHNMKOM4U57UVPQAAINTA, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.451821410 (536h40m56.322904445s) [noise=true , address=45.91.116.4:4940 , piece_id=VG6RXQRJTBM7663MNBCSGGUU7OR5KSJPK2ZJ3VOJZSUI2ZHF77CA, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.451848852 (536h40m56.322931887s) [noise=true , address=45.91.116.14:4191 , piece_id=I7VFSY2RP77AQO3VX4S5IAPMSMECRGTUG4B5FD3WPIEM7NJ3LP7A, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452139685 (536h40m56.323222721s) [noise=true , address=145.79.227.11:4731 , piece_id=Y6X7ISPGRDT63HU3UJJK7K4RLQVASDDYTNNWWMCGMU42N3J6CJAQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452167188 (536h40m56.323250223s) [noise=true , address=45.91.116.8:4047 , piece_id=FO7VXYZMLOD6HGJDDC7XW72NSPEYUKXHIP7UZTEL2T7BL22X4FGQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452392458 (536h40m56.323475503s) [noise=true , address=145.79.227.15:4876 , piece_id=5QS4QIX23NLQZOHFSL23QCDSZELHTM2A56YLKUEIUYYVP5XU4EDQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452484743 (536h40m56.323567778s) [noise=true , address=145.79.227.6:4258 , piece_id=WNIY2URYWLLFNWLWYINHQKBCZIPDNPIQTM6SVNY2BHNVT3EVOPIQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452655999 (536h40m56.323739034s) [noise=true , address=145.79.227.39:4013 , piece_id=BA76JFSGS3LTLF2XTSVPBMEX6BEPGSY2TZMWDEREMPTDHR6VJDGQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452805344 (536h40m56.323888389s) [noise=true , address=45.91.116.9:4262 , piece_id=RWDZUUSFQMEODBXZFS45RAG6VMO4RQHDR5XXOKTWRBEYENHLGAFQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452939859 (536h40m56.324022894s) [noise=true , address=45.91.116.11:4119 , piece_id=QIAZKVEZIYNDDARO6SDGDRZAAPZT7NV4KRPM35YQ2HT7BWUUHQWQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.453125714 (536h40m56.324208749s) [noise=true , address=77.90.31.13:4626 , piece_id=G6HEW6ESNVUAM2KZCZNRU3NBIO3QQJVC2JEYKQ636VQNXDQN5E4A, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.453206497 (536h40m56.324289532s) [noise=true , address=145.79.227.16:4911 , piece_id=4A7CMOANDRQSDO2IYTMTGLEAQTIG2MZ6R6PLSX34U6M33ADO23HA, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.453531746 (536h40m56.324614781s) [noise=true , address=145.79.227.9:4367 , piece_id=DLVKZJKAMUF2BWS7DW6RXL4H6LNYI2LN5KYQO35KUFVVTGGUKEAQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.453581532 (536h40m56.324664567s) [noise=true , address=145.79.227.17:4971 , piece_id=DTN63IW73RTCXGDJ5EUDMEXL2HCSHQWKAHJSFNTKKNYMNISIHH2Q, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:58.177129893 (536h40m58.048212938s) [upload=470674]: Next returned nil segment Dec 28 00:09:58.177144612 (536h40m58.048227647s) [upload=470674]: Waiting for error group managing segments... Dec 28 00:09:58.455943147 (536h40m58.327026193s) [noise=true , address=145.79.227.11:4731 , piece_id=Y6X7ISPGRDT63HU3UJJK7K4RLQVASDDYTNNWWMCGMU42N3J6CJAQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.456456094 (536h40m58.32753913s) [noise=true , address=145.79.227.15:4876 , piece_id=5QS4QIX23NLQZOHFSL23QCDSZELHTM2A56YLKUEIUYYVP5XU4EDQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.457186375 (536h40m58.32826942s) [noise=true , address=145.79.227.38:4979 , piece_id=IJNTI3OXIBUXPKSI2SBZX34UDSJPHWW7EHJSK3O546XLCMJHMQUA, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.468013020 (536h40m58.339096055s) [noise=true , address=145.79.227.9:4367 , piece_id=DLVKZJKAMUF2BWS7DW6RXL4H6LNYI2LN5KYQO35KUFVVTGGUKEAQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.480530786 (536h40m58.351613831s) [noise=true , address=145.79.227.16:4911 , piece_id=4A7CMOANDRQSDO2IYTMTGLEAQTIG2MZ6R6PLSX34U6M33ADO23HA, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.482180297 (536h40m58.353263332s) [noise=true , address=145.79.227.39:4013 , piece_id=BA76JFSGS3LTLF2XTSVPBMEX6BEPGSY2TZMWDEREMPTDHR6VJDGQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.489893524 (536h40m58.360976559s) [noise=true , address=145.79.227.51:4411 , piece_id=Z6Y26CT7UFG3L3YFGVHNCU76ND6IIVCXHNMKOM4U57UVPQAAINTA, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.545450293 (536h40m58.416533338s) [noise=true , address=145.79.227.17:4971 , piece_id=DTN63IW73RTCXGDJ5EUDMEXL2HCSHQWKAHJSFNTKKNYMNISIHH2Q, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.632175240 (536h40m58.503258285s) [noise=true , address=77.90.31.13:4626 , piece_id=G6HEW6ESNVUAM2KZCZNRU3NBIO3QQJVC2JEYKQ636VQNXDQN5E4A, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.670004554 (536h40m58.541087599s) [noise=true , address=45.91.116.8:4047 , piece_id=FO7VXYZMLOD6HGJDDC7XW72NSPEYUKXHIP7UZTEL2T7BL22X4FGQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.687035643 (536h40m58.558118688s) [noise=true , address=45.91.116.11:4119 , piece_id=QIAZKVEZIYNDDARO6SDGDRZAAPZT7NV4KRPM35YQ2HT7BWUUHQWQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.751210312 (536h40m58.622293357s) [noise=true , address=45.91.116.14:4191 , piece_id=I7VFSY2RP77AQO3VX4S5IAPMSMECRGTUG4B5FD3WPIEM7NJ3LP7A, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.762582436 (536h40m58.633665471s) [noise=true , address=45.91.116.9:4262 , piece_id=RWDZUUSFQMEODBXZFS45RAG6VMO4RQHDR5XXOKTWRBEYENHLGAFQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.762599809 (536h40m58.633682845s) [seg_pos={0 0}, upload=470674]: Segment reached optimal threshold of 13 pieces. Dec 28 00:09:58.762774011 (536h40m58.633857046s) [noise=true , address=145.79.227.6:4258 , piece_id=WNIY2URYWLLFNWLWYINHQKBCZIPDNPIQTM6SVNY2BHNVT3EVOPIQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:43566->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:43566->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43566->145.79.227.6:4258: use of closed network connection Dec 28 00:09:58.762779602 (536h40m58.633862647s) [noise=true , address=145.79.227.6:4258 , piece_id=WNIY2URYWLLFNWLWYINHQKBCZIPDNPIQTM6SVNY2BHNVT3EVOPIQ, seg_pos={0 0}, upload=470674]: Long tail cancellation due to optimal threshold Dec 28 00:09:58.763018958 (536h40m58.634102003s) [noise=true , address=145.79.227.41:4085 , piece_id=RIVEHAZY7SVH5YIWV3AVTN4FEFYLWPQVR25GPTWCW3ZU6DQUQ5QA, seg_pos={0 0}, upload=470674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 28 00:09:58.763024829 (536h40m58.634107874s) [noise=true , address=145.79.227.41:4085 , piece_id=RIVEHAZY7SVH5YIWV3AVTN4FEFYLWPQVR25GPTWCW3ZU6DQUQ5QA, seg_pos={0 0}, upload=470674]: Long tail cancellation due to optimal threshold Dec 28 00:09:58.763080525 (536h40m58.63416356s) [noise=true , address=45.91.116.4:4940 , piece_id=VG6RXQRJTBM7663MNBCSGGUU7OR5KSJPK2ZJ3VOJZSUI2ZHF77CA, seg_pos={0 0}, upload=470674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:39776->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:39776->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39776->45.91.116.4:4940: use of closed network connection Dec 28 00:09:58.763086997 (536h40m58.634170042s) [noise=true , address=45.91.116.4:4940 , piece_id=VG6RXQRJTBM7663MNBCSGGUU7OR5KSJPK2ZJ3VOJZSUI2ZHF77CA, seg_pos={0 0}, upload=470674]: Long tail cancellation due to optimal threshold Dec 28 00:09:58.763102847 (536h40m58.634185882s) [seg_pos={0 0}, upload=470674]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:09:58.763174283 (536h40m58.634257318s) [upload=470674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:09:58.763178421 (536h40m58.634261456s) [upload=470674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:09:59.047089331 (536h40m58.918172376s) [upload=470674]: Done uploading segments... Dec 28 00:09:59.047113126 (536h40m58.918196171s) [upload=470674]: Upload finished. err: <nil> Duration: 6.755754512s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.445957428s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:10:49 UTC) orbiter warning: context deadline exceeded Duration: 3.445988958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....X] duration: 30.010765763s trace: https://jaeger.dev.storj.io/trace/22b2b8c81bfa4fba FAIL (2025-12-28 00:10:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: QIAZKVEZIYNDDARO6SDGDRZAAPZT7NV4KRPM35YQ2HT7BWUUHQWQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: G6HEW6ESNVUAM2KZCZNRU3NBIO3QQJVC2JEYKQ636VQNXDQN5E4A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: FO7VXYZMLOD6HGJDDC7XW72NSPEYUKXHIP7UZTEL2T7BL22X4FGQ) context deadline exceeded ======LOGS====== ments... Dec 28 00:09:56.203334201 (536h40m56.074417246s) [upload=470674]: Got next segment. Inline: false Dec 28 00:09:56.203413443 (536h40m56.074496478s) [upload=470674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:09:56.203420416 (536h40m56.074503451s) [upload=470674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:09:56.450663154 (536h40m56.321746199s) [seg_pos={0 0}, upload=470674]: Begin upload segment... Dec 28 00:09:56.450704323 (536h40m56.321787358s) [seg_pos={0 0}, upload=470674]: Done begin upload segment. Dec 28 00:09:56.450736444 (536h40m56.321819489s) [noise=true , address=145.79.227.41:4085 , piece_id=RIVEHAZY7SVH5YIWV3AVTN4FEFYLWPQVR25GPTWCW3ZU6DQUQ5QA, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.450831355 (536h40m56.32191439s) [noise=true , address=145.79.227.38:4979 , piece_id=IJNTI3OXIBUXPKSI2SBZX34UDSJPHWW7EHJSK3O546XLCMJHMQUA, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.451450284 (536h40m56.322533339s) [noise=true , address=145.79.227.51:4411 , piece_id=Z6Y26CT7UFG3L3YFGVHNCU76ND6IIVCXHNMKOM4U57UVPQAAINTA, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.451821410 (536h40m56.322904445s) [noise=true , address=45.91.116.4:4940 , piece_id=VG6RXQRJTBM7663MNBCSGGUU7OR5KSJPK2ZJ3VOJZSUI2ZHF77CA, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.451848852 (536h40m56.322931887s) [noise=true , address=45.91.116.14:4191 , piece_id=I7VFSY2RP77AQO3VX4S5IAPMSMECRGTUG4B5FD3WPIEM7NJ3LP7A, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452139685 (536h40m56.323222721s) [noise=true , address=145.79.227.11:4731 , piece_id=Y6X7ISPGRDT63HU3UJJK7K4RLQVASDDYTNNWWMCGMU42N3J6CJAQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452167188 (536h40m56.323250223s) [noise=true , address=45.91.116.8:4047 , piece_id=FO7VXYZMLOD6HGJDDC7XW72NSPEYUKXHIP7UZTEL2T7BL22X4FGQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452392458 (536h40m56.323475503s) [noise=true , address=145.79.227.15:4876 , piece_id=5QS4QIX23NLQZOHFSL23QCDSZELHTM2A56YLKUEIUYYVP5XU4EDQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452484743 (536h40m56.323567778s) [noise=true , address=145.79.227.6:4258 , piece_id=WNIY2URYWLLFNWLWYINHQKBCZIPDNPIQTM6SVNY2BHNVT3EVOPIQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452655999 (536h40m56.323739034s) [noise=true , address=145.79.227.39:4013 , piece_id=BA76JFSGS3LTLF2XTSVPBMEX6BEPGSY2TZMWDEREMPTDHR6VJDGQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452805344 (536h40m56.323888389s) [noise=true , address=45.91.116.9:4262 , piece_id=RWDZUUSFQMEODBXZFS45RAG6VMO4RQHDR5XXOKTWRBEYENHLGAFQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.452939859 (536h40m56.324022894s) [noise=true , address=45.91.116.11:4119 , piece_id=QIAZKVEZIYNDDARO6SDGDRZAAPZT7NV4KRPM35YQ2HT7BWUUHQWQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.453125714 (536h40m56.324208749s) [noise=true , address=77.90.31.13:4626 , piece_id=G6HEW6ESNVUAM2KZCZNRU3NBIO3QQJVC2JEYKQ636VQNXDQN5E4A, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.453206497 (536h40m56.324289532s) [noise=true , address=145.79.227.16:4911 , piece_id=4A7CMOANDRQSDO2IYTMTGLEAQTIG2MZ6R6PLSX34U6M33ADO23HA, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.453531746 (536h40m56.324614781s) [noise=true , address=145.79.227.9:4367 , piece_id=DLVKZJKAMUF2BWS7DW6RXL4H6LNYI2LN5KYQO35KUFVVTGGUKEAQ, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:56.453581532 (536h40m56.324664567s) [noise=true , address=145.79.227.17:4971 , piece_id=DTN63IW73RTCXGDJ5EUDMEXL2HCSHQWKAHJSFNTKKNYMNISIHH2Q, seg_pos={0 0}, upload=470674]: Uploading piece... Dec 28 00:09:58.177129893 (536h40m58.048212938s) [upload=470674]: Next returned nil segment Dec 28 00:09:58.177144612 (536h40m58.048227647s) [upload=470674]: Waiting for error group managing segments... Dec 28 00:09:58.455943147 (536h40m58.327026193s) [noise=true , address=145.79.227.11:4731 , piece_id=Y6X7ISPGRDT63HU3UJJK7K4RLQVASDDYTNNWWMCGMU42N3J6CJAQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.456456094 (536h40m58.32753913s) [noise=true , address=145.79.227.15:4876 , piece_id=5QS4QIX23NLQZOHFSL23QCDSZELHTM2A56YLKUEIUYYVP5XU4EDQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.457186375 (536h40m58.32826942s) [noise=true , address=145.79.227.38:4979 , piece_id=IJNTI3OXIBUXPKSI2SBZX34UDSJPHWW7EHJSK3O546XLCMJHMQUA, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.468013020 (536h40m58.339096055s) [noise=true , address=145.79.227.9:4367 , piece_id=DLVKZJKAMUF2BWS7DW6RXL4H6LNYI2LN5KYQO35KUFVVTGGUKEAQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.480530786 (536h40m58.351613831s) [noise=true , address=145.79.227.16:4911 , piece_id=4A7CMOANDRQSDO2IYTMTGLEAQTIG2MZ6R6PLSX34U6M33ADO23HA, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.482180297 (536h40m58.353263332s) [noise=true , address=145.79.227.39:4013 , piece_id=BA76JFSGS3LTLF2XTSVPBMEX6BEPGSY2TZMWDEREMPTDHR6VJDGQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.489893524 (536h40m58.360976559s) [noise=true , address=145.79.227.51:4411 , piece_id=Z6Y26CT7UFG3L3YFGVHNCU76ND6IIVCXHNMKOM4U57UVPQAAINTA, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.545450293 (536h40m58.416533338s) [noise=true , address=145.79.227.17:4971 , piece_id=DTN63IW73RTCXGDJ5EUDMEXL2HCSHQWKAHJSFNTKKNYMNISIHH2Q, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.632175240 (536h40m58.503258285s) [noise=true , address=77.90.31.13:4626 , piece_id=G6HEW6ESNVUAM2KZCZNRU3NBIO3QQJVC2JEYKQ636VQNXDQN5E4A, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.670004554 (536h40m58.541087599s) [noise=true , address=45.91.116.8:4047 , piece_id=FO7VXYZMLOD6HGJDDC7XW72NSPEYUKXHIP7UZTEL2T7BL22X4FGQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.687035643 (536h40m58.558118688s) [noise=true , address=45.91.116.11:4119 , piece_id=QIAZKVEZIYNDDARO6SDGDRZAAPZT7NV4KRPM35YQ2HT7BWUUHQWQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.751210312 (536h40m58.622293357s) [noise=true , address=45.91.116.14:4191 , piece_id=I7VFSY2RP77AQO3VX4S5IAPMSMECRGTUG4B5FD3WPIEM7NJ3LP7A, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.762582436 (536h40m58.633665471s) [noise=true , address=45.91.116.9:4262 , piece_id=RWDZUUSFQMEODBXZFS45RAG6VMO4RQHDR5XXOKTWRBEYENHLGAFQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: <nil> Dec 28 00:09:58.762599809 (536h40m58.633682845s) [seg_pos={0 0}, upload=470674]: Segment reached optimal threshold of 13 pieces. Dec 28 00:09:58.762774011 (536h40m58.633857046s) [noise=true , address=145.79.227.6:4258 , piece_id=WNIY2URYWLLFNWLWYINHQKBCZIPDNPIQTM6SVNY2BHNVT3EVOPIQ, seg_pos={0 0}, upload=470674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:43566->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:43566->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43566->145.79.227.6:4258: use of closed network connection Dec 28 00:09:58.762779602 (536h40m58.633862647s) [noise=true , address=145.79.227.6:4258 , piece_id=WNIY2URYWLLFNWLWYINHQKBCZIPDNPIQTM6SVNY2BHNVT3EVOPIQ, seg_pos={0 0}, upload=470674]: Long tail cancellation due to optimal threshold Dec 28 00:09:58.763018958 (536h40m58.634102003s) [noise=true , address=145.79.227.41:4085 , piece_id=RIVEHAZY7SVH5YIWV3AVTN4FEFYLWPQVR25GPTWCW3ZU6DQUQ5QA, seg_pos={0 0}, upload=470674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 28 00:09:58.763024829 (536h40m58.634107874s) [noise=true , address=145.79.227.41:4085 , piece_id=RIVEHAZY7SVH5YIWV3AVTN4FEFYLWPQVR25GPTWCW3ZU6DQUQ5QA, seg_pos={0 0}, upload=470674]: Long tail cancellation due to optimal threshold Dec 28 00:09:58.763080525 (536h40m58.63416356s) [noise=true , address=45.91.116.4:4940 , piece_id=VG6RXQRJTBM7663MNBCSGGUU7OR5KSJPK2ZJ3VOJZSUI2ZHF77CA, seg_pos={0 0}, upload=470674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:39776->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:39776->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39776->45.91.116.4:4940: use of closed network connection Dec 28 00:09:58.763086997 (536h40m58.634170042s) [noise=true , address=45.91.116.4:4940 , piece_id=VG6RXQRJTBM7663MNBCSGGUU7OR5KSJPK2ZJ3VOJZSUI2ZHF77CA, seg_pos={0 0}, upload=470674]: Long tail cancellation due to optimal threshold Dec 28 00:09:58.763102847 (536h40m58.634185882s) [seg_pos={0 0}, upload=470674]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:09:58.763174283 (536h40m58.634257318s) [upload=470674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:09:58.763178421 (536h40m58.634261456s) [upload=470674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:09:59.047089331 (536h40m58.918172376s) [upload=470674]: Done uploading segments... Dec 28 00:09:59.047113126 (536h40m58.918196171s) [upload=470674]: Upload finished. err: <nil> Duration: 30.010779718s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.497346218s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:10:09 UTC) orbiter warning: context deadline exceeded Duration: 3.497499481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X....] duration: 24.605040992s trace: https://jaeger.dev.storj.io/trace/384c4d40af21c95 FAIL (2025-12-28 00:09:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA) context deadline exceeded ======LOGS====== Dec 28 00:06:58.755643965 (536h37m58.626727s) [upload=470630]: Starting upload Dec 28 00:06:58.755730780 (536h37m58.626813825s) [upload=470630]: Done starting upload Dec 28 00:06:58.755962872 (536h37m58.627045917s) [upload=470630]: Uploading segments... Dec 28 00:06:58.756016864 (536h37m58.627099909s) [upload=470630]: Got next segment. Inline: false Dec 28 00:06:58.756052443 (536h37m58.627135488s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:06:58.756058484 (536h37m58.627141529s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:06:59.002925186 (536h37m58.874008251s) [seg_pos={0 0}, upload=470630]: Begin upload segment... Dec 28 00:06:59.003003806 (536h37m58.874086851s) [seg_pos={0 0}, upload=470630]: Done begin upload segment. Dec 28 00:06:59.003086232 (536h37m58.874169287s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003110940 (536h37m58.874193985s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003741070 (536h37m58.874824115s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004179204 (536h37m58.875262249s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004822990 (536h37m58.875906045s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005348981 (536h37m58.876432046s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005990313 (536h37m58.877073358s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006605795 (536h37m58.87768884s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006661481 (536h37m58.877744636s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007106849 (536h37m58.878189884s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007433751 (536h37m58.878516796s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007597433 (536h37m58.878680468s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007974882 (536h37m58.879057927s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008080393 (536h37m58.879163448s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008696085 (536h37m58.87977913s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.009199393 (536h37m58.880282438s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:07:00.629723812 (536h38m0.500806867s) [upload=470630]: Next returned nil segment Dec 28 00:07:00.629747377 (536h38m0.500830422s) [upload=470630]: Waiting for error group managing segments... Dec 28 00:07:01.013634658 (536h38m0.884717703s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015746799 (536h38m0.886829834s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015865726 (536h38m0.886948762s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.016264596 (536h38m0.887347631s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018397958 (536h38m0.889481003s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018551210 (536h38m0.889634255s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.029408714 (536h38m0.900491749s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.038087430 (536h38m0.909170475s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.176891274 (536h38m1.047974319s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.192264767 (536h38m1.063347802s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.193100879 (536h38m1.064183924s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.241952340 (536h38m1.113035385s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267854892 (536h38m1.138937938s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267878707 (536h38m1.138961752s) [seg_pos={0 0}, upload=470630]: Segment reached optimal threshold of 13 pieces. Dec 28 00:07:01.268113044 (536h38m1.139196089s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268121159 (536h38m1.139204204s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268269722 (536h38m1.139352768s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268276916 (536h38m1.139359961s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268521391 (536h38m1.139604427s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268539165 (536h38m1.1396222s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268563572 (536h38m1.139646607s) [seg_pos={0 0}, upload=470630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:07:01.268652241 (536h38m1.139735276s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:07:01.268658332 (536h38m1.139741377s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:07:01.567034296 (536h38m1.438117341s) [upload=470630]: Done uploading segments... Dec 28 00:07:01.567062038 (536h38m1.438145083s) [upload=470630]: Upload finished. err: <nil> Duration: 24.605065188s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.401297696s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:09:43 UTC) orbiter warning: context deadline exceeded Duration: 3.40132589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.447839676s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:08:49 UTC) orbiter warning: context deadline exceeded Duration: 3.447865577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X...] duration: 5.973833756s trace: https://jaeger.dev.storj.io/trace/384c4d40af21c95 FAIL (2025-12-28 00:08:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA) context deadline exceeded ======LOGS====== Dec 28 00:06:58.755643965 (536h37m58.626727s) [upload=470630]: Starting upload Dec 28 00:06:58.755730780 (536h37m58.626813825s) [upload=470630]: Done starting upload Dec 28 00:06:58.755962872 (536h37m58.627045917s) [upload=470630]: Uploading segments... Dec 28 00:06:58.756016864 (536h37m58.627099909s) [upload=470630]: Got next segment. Inline: false Dec 28 00:06:58.756052443 (536h37m58.627135488s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:06:58.756058484 (536h37m58.627141529s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:06:59.002925186 (536h37m58.874008251s) [seg_pos={0 0}, upload=470630]: Begin upload segment... Dec 28 00:06:59.003003806 (536h37m58.874086851s) [seg_pos={0 0}, upload=470630]: Done begin upload segment. Dec 28 00:06:59.003086232 (536h37m58.874169287s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003110940 (536h37m58.874193985s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003741070 (536h37m58.874824115s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004179204 (536h37m58.875262249s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004822990 (536h37m58.875906045s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005348981 (536h37m58.876432046s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005990313 (536h37m58.877073358s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006605795 (536h37m58.87768884s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006661481 (536h37m58.877744636s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007106849 (536h37m58.878189884s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007433751 (536h37m58.878516796s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007597433 (536h37m58.878680468s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007974882 (536h37m58.879057927s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008080393 (536h37m58.879163448s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008696085 (536h37m58.87977913s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.009199393 (536h37m58.880282438s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:07:00.629723812 (536h38m0.500806867s) [upload=470630]: Next returned nil segment Dec 28 00:07:00.629747377 (536h38m0.500830422s) [upload=470630]: Waiting for error group managing segments... Dec 28 00:07:01.013634658 (536h38m0.884717703s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015746799 (536h38m0.886829834s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015865726 (536h38m0.886948762s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.016264596 (536h38m0.887347631s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018397958 (536h38m0.889481003s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018551210 (536h38m0.889634255s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.029408714 (536h38m0.900491749s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.038087430 (536h38m0.909170475s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.176891274 (536h38m1.047974319s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.192264767 (536h38m1.063347802s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.193100879 (536h38m1.064183924s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.241952340 (536h38m1.113035385s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267854892 (536h38m1.138937938s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267878707 (536h38m1.138961752s) [seg_pos={0 0}, upload=470630]: Segment reached optimal threshold of 13 pieces. Dec 28 00:07:01.268113044 (536h38m1.139196089s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268121159 (536h38m1.139204204s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268269722 (536h38m1.139352768s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268276916 (536h38m1.139359961s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268521391 (536h38m1.139604427s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268539165 (536h38m1.1396222s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268563572 (536h38m1.139646607s) [seg_pos={0 0}, upload=470630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:07:01.268652241 (536h38m1.139735276s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:07:01.268658332 (536h38m1.139741377s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:07:01.567034296 (536h38m1.438117341s) [upload=470630]: Done uploading segments... Dec 28 00:07:01.567062038 (536h38m1.438145083s) [upload=470630]: Upload finished. err: <nil> Duration: 5.973866808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X..] duration: 5.841782389s trace: https://jaeger.dev.storj.io/trace/384c4d40af21c95 FAIL (2025-12-28 00:08:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA) context deadline exceeded ======LOGS====== Dec 28 00:06:58.755643965 (536h37m58.626727s) [upload=470630]: Starting upload Dec 28 00:06:58.755730780 (536h37m58.626813825s) [upload=470630]: Done starting upload Dec 28 00:06:58.755962872 (536h37m58.627045917s) [upload=470630]: Uploading segments... Dec 28 00:06:58.756016864 (536h37m58.627099909s) [upload=470630]: Got next segment. Inline: false Dec 28 00:06:58.756052443 (536h37m58.627135488s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:06:58.756058484 (536h37m58.627141529s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:06:59.002925186 (536h37m58.874008251s) [seg_pos={0 0}, upload=470630]: Begin upload segment... Dec 28 00:06:59.003003806 (536h37m58.874086851s) [seg_pos={0 0}, upload=470630]: Done begin upload segment. Dec 28 00:06:59.003086232 (536h37m58.874169287s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003110940 (536h37m58.874193985s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003741070 (536h37m58.874824115s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004179204 (536h37m58.875262249s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004822990 (536h37m58.875906045s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005348981 (536h37m58.876432046s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005990313 (536h37m58.877073358s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006605795 (536h37m58.87768884s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006661481 (536h37m58.877744636s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007106849 (536h37m58.878189884s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007433751 (536h37m58.878516796s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007597433 (536h37m58.878680468s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007974882 (536h37m58.879057927s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008080393 (536h37m58.879163448s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008696085 (536h37m58.87977913s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.009199393 (536h37m58.880282438s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:07:00.629723812 (536h38m0.500806867s) [upload=470630]: Next returned nil segment Dec 28 00:07:00.629747377 (536h38m0.500830422s) [upload=470630]: Waiting for error group managing segments... Dec 28 00:07:01.013634658 (536h38m0.884717703s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015746799 (536h38m0.886829834s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015865726 (536h38m0.886948762s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.016264596 (536h38m0.887347631s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018397958 (536h38m0.889481003s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018551210 (536h38m0.889634255s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.029408714 (536h38m0.900491749s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.038087430 (536h38m0.909170475s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.176891274 (536h38m1.047974319s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.192264767 (536h38m1.063347802s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.193100879 (536h38m1.064183924s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.241952340 (536h38m1.113035385s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267854892 (536h38m1.138937938s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267878707 (536h38m1.138961752s) [seg_pos={0 0}, upload=470630]: Segment reached optimal threshold of 13 pieces. Dec 28 00:07:01.268113044 (536h38m1.139196089s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268121159 (536h38m1.139204204s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268269722 (536h38m1.139352768s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268276916 (536h38m1.139359961s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268521391 (536h38m1.139604427s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268539165 (536h38m1.1396222s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268563572 (536h38m1.139646607s) [seg_pos={0 0}, upload=470630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:07:01.268652241 (536h38m1.139735276s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:07:01.268658332 (536h38m1.139741377s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:07:01.567034296 (536h38m1.438117341s) [upload=470630]: Done uploading segments... Dec 28 00:07:01.567062038 (536h38m1.438145083s) [upload=470630]: Upload finished. err: <nil> Duration: 5.841807978s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.516744566s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:08:15 UTC) orbiter warning: context deadline exceeded Duration: 3.516767019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X.] duration: 7.052989103s trace: https://jaeger.dev.storj.io/trace/384c4d40af21c95 FAIL (2025-12-28 00:07:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA) context deadline exceeded ======LOGS====== Dec 28 00:06:58.755643965 (536h37m58.626727s) [upload=470630]: Starting upload Dec 28 00:06:58.755730780 (536h37m58.626813825s) [upload=470630]: Done starting upload Dec 28 00:06:58.755962872 (536h37m58.627045917s) [upload=470630]: Uploading segments... Dec 28 00:06:58.756016864 (536h37m58.627099909s) [upload=470630]: Got next segment. Inline: false Dec 28 00:06:58.756052443 (536h37m58.627135488s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:06:58.756058484 (536h37m58.627141529s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:06:59.002925186 (536h37m58.874008251s) [seg_pos={0 0}, upload=470630]: Begin upload segment... Dec 28 00:06:59.003003806 (536h37m58.874086851s) [seg_pos={0 0}, upload=470630]: Done begin upload segment. Dec 28 00:06:59.003086232 (536h37m58.874169287s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003110940 (536h37m58.874193985s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003741070 (536h37m58.874824115s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004179204 (536h37m58.875262249s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004822990 (536h37m58.875906045s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005348981 (536h37m58.876432046s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005990313 (536h37m58.877073358s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006605795 (536h37m58.87768884s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006661481 (536h37m58.877744636s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007106849 (536h37m58.878189884s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007433751 (536h37m58.878516796s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007597433 (536h37m58.878680468s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007974882 (536h37m58.879057927s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008080393 (536h37m58.879163448s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008696085 (536h37m58.87977913s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.009199393 (536h37m58.880282438s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:07:00.629723812 (536h38m0.500806867s) [upload=470630]: Next returned nil segment Dec 28 00:07:00.629747377 (536h38m0.500830422s) [upload=470630]: Waiting for error group managing segments... Dec 28 00:07:01.013634658 (536h38m0.884717703s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015746799 (536h38m0.886829834s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015865726 (536h38m0.886948762s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.016264596 (536h38m0.887347631s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018397958 (536h38m0.889481003s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018551210 (536h38m0.889634255s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.029408714 (536h38m0.900491749s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.038087430 (536h38m0.909170475s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.176891274 (536h38m1.047974319s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.192264767 (536h38m1.063347802s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.193100879 (536h38m1.064183924s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.241952340 (536h38m1.113035385s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267854892 (536h38m1.138937938s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267878707 (536h38m1.138961752s) [seg_pos={0 0}, upload=470630]: Segment reached optimal threshold of 13 pieces. Dec 28 00:07:01.268113044 (536h38m1.139196089s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268121159 (536h38m1.139204204s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268269722 (536h38m1.139352768s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268276916 (536h38m1.139359961s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268521391 (536h38m1.139604427s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268539165 (536h38m1.1396222s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268563572 (536h38m1.139646607s) [seg_pos={0 0}, upload=470630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:07:01.268652241 (536h38m1.139735276s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:07:01.268658332 (536h38m1.139741377s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:07:01.567034296 (536h38m1.438117341s) [upload=470630]: Done uploading segments... Dec 28 00:07:01.567062038 (536h38m1.438145083s) [upload=470630]: Upload finished. err: <nil> Duration: 7.053007798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....X] duration: 30.002541904s trace: https://jaeger.dev.storj.io/trace/384c4d40af21c95 FAIL (2025-12-28 00:07:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA) context deadline exceeded ======LOGS====== Dec 28 00:06:58.755643965 (536h37m58.626727s) [upload=470630]: Starting upload Dec 28 00:06:58.755730780 (536h37m58.626813825s) [upload=470630]: Done starting upload Dec 28 00:06:58.755962872 (536h37m58.627045917s) [upload=470630]: Uploading segments... Dec 28 00:06:58.756016864 (536h37m58.627099909s) [upload=470630]: Got next segment. Inline: false Dec 28 00:06:58.756052443 (536h37m58.627135488s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:06:58.756058484 (536h37m58.627141529s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:06:59.002925186 (536h37m58.874008251s) [seg_pos={0 0}, upload=470630]: Begin upload segment... Dec 28 00:06:59.003003806 (536h37m58.874086851s) [seg_pos={0 0}, upload=470630]: Done begin upload segment. Dec 28 00:06:59.003086232 (536h37m58.874169287s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003110940 (536h37m58.874193985s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.003741070 (536h37m58.874824115s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004179204 (536h37m58.875262249s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.004822990 (536h37m58.875906045s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005348981 (536h37m58.876432046s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.005990313 (536h37m58.877073358s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006605795 (536h37m58.87768884s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.006661481 (536h37m58.877744636s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007106849 (536h37m58.878189884s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007433751 (536h37m58.878516796s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007597433 (536h37m58.878680468s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.007974882 (536h37m58.879057927s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008080393 (536h37m58.879163448s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.008696085 (536h37m58.87977913s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:06:59.009199393 (536h37m58.880282438s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Uploading piece... Dec 28 00:07:00.629723812 (536h38m0.500806867s) [upload=470630]: Next returned nil segment Dec 28 00:07:00.629747377 (536h38m0.500830422s) [upload=470630]: Waiting for error group managing segments... Dec 28 00:07:01.013634658 (536h38m0.884717703s) [noise=true , address=145.79.227.48:4303 , piece_id=257QC2A7IAKTXL3TAOPD67I7HAEKRGD6WOQH74LSBF2D3MMDZ65A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015746799 (536h38m0.886829834s) [noise=true , address=145.79.227.11:4731 , piece_id=KYEIT5MHPFUIFAB7EXLHBCZTRIVBFKC566USSZG222RVAXJBUKEA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.015865726 (536h38m0.886948762s) [noise=true , address=145.79.227.5:4221 , piece_id=VCYXGAQAQLSGEEPNK3W2XWMJGAQQL55EGN5OYRQSCSX2FSR35LQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.016264596 (536h38m0.887347631s) [noise=true , address=145.79.227.9:4365 , piece_id=GCEZADFWDEJWICUKK2HBS5325S2M7ILV6YFQIO7PCEIPMOFFUMBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018397958 (536h38m0.889481003s) [noise=true , address=145.79.227.38:4978 , piece_id=4YUI47Z4ORYRXLQLRA465DFY4RDFGNHZI6AZPCZHSVSVJ5C2GBCA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.018551210 (536h38m0.889634255s) [noise=true , address=145.79.227.42:4121 , piece_id=WWPNGJBJQSED3JHTQJTXGZWAMAJG4WLS3463FI6YI4WZNXN4GRBQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.029408714 (536h38m0.900491749s) [noise=true , address=145.79.227.39:4015 , piece_id=372QRMUADC73ATDJZM6MLXRTBY2DZ6ERE7TRE44ZH6LG7XYGR5YQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.038087430 (536h38m0.909170475s) [noise=true , address=145.79.227.45:4230 , piece_id=H5YIN3KC5I6MWJ3GA67D733JN3ZJT3LZI4XFGGLA53MHNGGNYYJQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.176891274 (536h38m1.047974319s) [noise=true , address=77.90.31.13:4627 , piece_id=7X7BOQJ5HTKBO6SKU2WZ6ALJEPN2AZSVSTDKPE63OUFJAWRUL5CQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.192264767 (536h38m1.063347802s) [noise=true , address=77.90.31.16:4736 , piece_id=AZDR5V5MKKZUL5VZSW7XKMHCHGE7ZVOFTNM5ADSQY37MJ3HJIDQQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.193100879 (536h38m1.064183924s) [noise=true , address=77.90.31.11:4553 , piece_id=64HWAMUGVSKTL54ASELPSEBLJK47M4XOH44F3WNW5N5EVOTPVL2A, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.241952340 (536h38m1.113035385s) [noise=true , address=45.91.116.13:4156 , piece_id=6BCU3C2U7EXT3CPOKTIQ3EHNP2HQKYNVVRK4KJ7IDTMQ7IU46GFA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267854892 (536h38m1.138937938s) [noise=true , address=45.91.116.9:4262 , piece_id=AXA6RGLAMKLYY7YOTAEA4P7RCCGFYSNAKTFKSGQ3CWN4KZVYE2XQ, seg_pos={0 0}, upload=470630]: Done uploading piece. err: <nil> Dec 28 00:07:01.267878707 (536h38m1.138961752s) [seg_pos={0 0}, upload=470630]: Segment reached optimal threshold of 13 pieces. Dec 28 00:07:01.268113044 (536h38m1.139196089s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268121159 (536h38m1.139204204s) [noise=true , address=145.79.227.35:4483 , piece_id=IPT4C4M4B52NOYU4OELNJHQST4HEGJJMAPDRQWGXD6V3BOQZ5JHA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268269722 (536h38m1.139352768s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268276916 (536h38m1.139359961s) [noise=true , address=145.79.227.12:4765 , piece_id=FN6MRNK3PXS646HXODGEZGEEXQUEGH53AWUV6ZO5IOR5CDFIJM5Q, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268521391 (536h38m1.139604427s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:07:01.268539165 (536h38m1.1396222s) [noise=true , address=77.90.31.21:4878 , piece_id=3UEY47F26YIBLEK5HTQWVNCFYIJH5L5IBGY4G662T4NJHLVQCCQA, seg_pos={0 0}, upload=470630]: Long tail cancellation due to optimal threshold Dec 28 00:07:01.268563572 (536h38m1.139646607s) [seg_pos={0 0}, upload=470630]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:07:01.268652241 (536h38m1.139735276s) [upload=470630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:07:01.268658332 (536h38m1.139741377s) [upload=470630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:07:01.567034296 (536h38m1.438117341s) [upload=470630]: Done uploading segments... Dec 28 00:07:01.567062038 (536h38m1.438145083s) [upload=470630]: Upload finished. err: <nil> Duration: 30.002563476s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.425485202s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:07:11 UTC) orbiter warning: context deadline exceeded Duration: 3.42552085s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X....] duration: 6.045468522s trace: https://jaeger.dev.storj.io/trace/6ecb5d8fd65b494 FAIL (2025-12-28 00:06:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA) context deadline exceeded ======LOGS====== Dec 28 00:02:46.568114009 (536h33m46.439197044s) [upload=470567]: Starting upload Dec 28 00:02:46.568209140 (536h33m46.439292185s) [upload=470567]: Done starting upload Dec 28 00:02:46.568397779 (536h33m46.439480825s) [upload=470567]: Uploading segments... Dec 28 00:02:46.568446331 (536h33m46.439529377s) [upload=470567]: Got next segment. Inline: false Dec 28 00:02:46.568483823 (536h33m46.439566878s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:02:46.568488612 (536h33m46.439571657s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:02:46.816788503 (536h33m46.687871548s) [seg_pos={0 0}, upload=470567]: Begin upload segment... Dec 28 00:02:46.816835762 (536h33m46.687918808s) [seg_pos={0 0}, upload=470567]: Done begin upload segment. Dec 28 00:02:46.816861922 (536h33m46.687944968s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.816912359 (536h33m46.687995404s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.817511941 (536h33m46.688594986s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818001342 (536h33m46.689084387s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818548324 (536h33m46.689631369s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818840911 (536h33m46.689923946s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819109232 (536h33m46.690192267s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819376210 (536h33m46.690459245s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819754300 (536h33m46.690837345s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.820802195 (536h33m46.69188523s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822382525 (536h33m46.69346556s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822467296 (536h33m46.693550331s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.823907859 (536h33m46.694990894s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.825278720 (536h33m46.696361755s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.826564709 (536h33m46.697647744s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.827946910 (536h33m46.699029955s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:48.311283783 (536h33m48.182366828s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: not enough available disk space, have: 0, need: 9607936 Dec 28 00:02:48.503370406 (536h33m48.374453441s) [upload=470567]: Next returned nil segment Dec 28 00:02:48.503382418 (536h33m48.374465454s) [upload=470567]: Waiting for error group managing segments... Dec 28 00:02:48.820744675 (536h33m48.69182774s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.839365583 (536h33m48.710448618s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.841998577 (536h33m48.713081612s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844333131 (536h33m48.715416166s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844817053 (536h33m48.715900098s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846458097 (536h33m48.717541132s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846857288 (536h33m48.717940333s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846927682 (536h33m48.718010727s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.852120159 (536h33m48.723203194s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.019732088 (536h33m48.890815123s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.082157426 (536h33m48.953240461s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.091696009 (536h33m48.962779044s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149354459 (536h33m49.020437504s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149380078 (536h33m49.020463123s) [seg_pos={0 0}, upload=470567]: Segment reached optimal threshold of 13 pieces. Dec 28 00:02:49.149742890 (536h33m49.020825935s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 00:02:49.149753870 (536h33m49.020836905s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149872055 (536h33m49.02095509s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection Dec 28 00:02:49.149898205 (536h33m49.02098125s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149957728 (536h33m49.021040774s) [seg_pos={0 0}, upload=470567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:02:49.150110519 (536h33m49.021193564s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:02:49.150116971 (536h33m49.021200016s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:02:49.438826439 (536h33m49.309909474s) [upload=470567]: Done uploading segments... Dec 28 00:02:49.438841117 (536h33m49.309924152s) [upload=470567]: Upload finished. err: <nil> Duration: 6.045484194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.....XX.XX.XXXX.XXXXXXX.X...X...XXX.XX..XX.XXX....X.X...X...] duration: 26.306514152s trace: https://jaeger.dev.storj.io/trace/6ecb5d8fd65b494 FAIL (2025-12-28 00:06:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA) context deadline exceeded ======LOGS====== Dec 28 00:02:46.568114009 (536h33m46.439197044s) [upload=470567]: Starting upload Dec 28 00:02:46.568209140 (536h33m46.439292185s) [upload=470567]: Done starting upload Dec 28 00:02:46.568397779 (536h33m46.439480825s) [upload=470567]: Uploading segments... Dec 28 00:02:46.568446331 (536h33m46.439529377s) [upload=470567]: Got next segment. Inline: false Dec 28 00:02:46.568483823 (536h33m46.439566878s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:02:46.568488612 (536h33m46.439571657s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:02:46.816788503 (536h33m46.687871548s) [seg_pos={0 0}, upload=470567]: Begin upload segment... Dec 28 00:02:46.816835762 (536h33m46.687918808s) [seg_pos={0 0}, upload=470567]: Done begin upload segment. Dec 28 00:02:46.816861922 (536h33m46.687944968s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.816912359 (536h33m46.687995404s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.817511941 (536h33m46.688594986s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818001342 (536h33m46.689084387s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818548324 (536h33m46.689631369s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.818840911 (536h33m46.689923946s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819109232 (536h33m46.690192267s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819376210 (536h33m46.690459245s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.819754300 (536h33m46.690837345s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.820802195 (536h33m46.69188523s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822382525 (536h33m46.69346556s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.822467296 (536h33m46.693550331s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.823907859 (536h33m46.694990894s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.825278720 (536h33m46.696361755s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.826564709 (536h33m46.697647744s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:46.827946910 (536h33m46.699029955s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Uploading piece... Dec 28 00:02:48.311283783 (536h33m48.182366828s) [noise=true , address=145.79.227.6:4259 , piece_id=NP3G7KMSRZHD52JDU4HM4QU3MQAVEE352SFXV2J76LPVEL4AIYVQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: not enough available disk space, have: 0, need: 9607936 Dec 28 00:02:48.503370406 (536h33m48.374453441s) [upload=470567]: Next returned nil segment Dec 28 00:02:48.503382418 (536h33m48.374465454s) [upload=470567]: Waiting for error group managing segments... Dec 28 00:02:48.820744675 (536h33m48.69182774s) [noise=true , address=145.79.227.37:4566 , piece_id=4SUUUVGPRK7ZNJ5YWAYZLUJ654LKHTAEWGLNEODBOTE7FML3RKEA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.839365583 (536h33m48.710448618s) [noise=true , address=145.79.227.9:4367 , piece_id=276I6QHCWRZMN5K5KGVT3LMJEYYPTQBYIYBLUGIINO4XAZNUQDSA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.841998577 (536h33m48.713081612s) [noise=true , address=145.79.227.18:4006 , piece_id=WMQXZSZEMRY3ESE6YWUD44SEXTAVJ3LRQXLSJWEWZIITXI4ZHNZA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844333131 (536h33m48.715416166s) [noise=true , address=145.79.227.38:4977 , piece_id=GONSHZVGHW72UDXZYGCRXFBISAK6EHZEYCF72YZUJT35WQHUTOPQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.844817053 (536h33m48.715900098s) [noise=true , address=145.79.227.5:4221 , piece_id=65H534IDXTRYAA67DPWB5SWO7P4QA4JP43NISZOX2FD2XZEKT4BQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846458097 (536h33m48.717541132s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AVKEG4LJ7NMDVYR23ZBC3TJUMOXGPOAQN5Q23UFEUI4X2F6C6A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846857288 (536h33m48.717940333s) [noise=true , address=145.79.227.12:4765 , piece_id=WMJX3BIM3JXT3ZGQ3PQE5P7R77P4RNYYQOBE7REEPZE42FWBCFQQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.846927682 (536h33m48.718010727s) [noise=true , address=145.79.227.46:4603 , piece_id=UXEUD4DJCTFGWTDKMIL2ZLI2LPETU2L3L63N7VC7MWACGZCA2ZEQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:48.852120159 (536h33m48.723203194s) [noise=true , address=145.79.227.36:4519 , piece_id=P26EIOLITWJTQUQ2YTWCR6BXT6YKN3VV7U2MZYBIJCPWHUU2F7TA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.019732088 (536h33m48.890815123s) [noise=true , address=77.90.31.13:4629 , piece_id=XZ6ZUSYVHLGT4B5VSYEQAC2VXJTA3DNOM55PL6W5XA2USP4XCHMQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.082157426 (536h33m48.953240461s) [noise=true , address=77.90.31.14:4664 , piece_id=OGPJWWYCYOUNI3OE3FM2WFA452YURX4LLUNVRI4MKP4JUCOKG2QQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.091696009 (536h33m48.962779044s) [noise=true , address=45.91.116.13:4155 , piece_id=XQL3MD42H5QDYXFSKULK5UKFUHG7TBAM6CK3LMOSPHJM53NVSCXQ, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149354459 (536h33m49.020437504s) [noise=true , address=45.91.116.12:4226 , piece_id=QGY2LWLBDXJHXV33J4CMBKZNCSUITVEFSBGFT7S6SX6PMYZLTBWA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: <nil> Dec 28 00:02:49.149380078 (536h33m49.020463123s) [seg_pos={0 0}, upload=470567]: Segment reached optimal threshold of 13 pieces. Dec 28 00:02:49.149742890 (536h33m49.020825935s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 00:02:49.149753870 (536h33m49.020836905s) [noise=true , address=77.90.31.15:4700 , piece_id=UHRITGCYWZA2NU3Z5P6KUC7L2MDI6BKUMO7QBLCHTW5OA62DOPUA, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149872055 (536h33m49.02095509s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48720->77.90.31.11:4556: use of closed network connection Dec 28 00:02:49.149898205 (536h33m49.02098125s) [noise=true , address=77.90.31.11:4556 , piece_id=NXWBSLCPUA6F6NQXUCOBOQMKQT5P2JWSUIWACRBC3GFXIQAKGP2A, seg_pos={0 0}, upload=470567]: Long tail cancellation due to optimal threshold Dec 28 00:02:49.149957728 (536h33m49.021040774s) [seg_pos={0 0}, upload=470567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:02:49.150110519 (536h33m49.021193564s) [upload=470567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:02:49.150116971 (536h33m49.021200016s) [upload=470567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:02:49.438826439 (536h33m49.309909474s) [upload=470567]: Done uploading segments... Dec 28 00:02:49.438841117 (536h33m49.309924152s) [upload=470567]: Upload finished. err: <nil> Duration: 26.306549269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.531660249s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:06:11 UTC) orbiter warning: context deadline exceeded Duration: 3.531690128s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.542838604s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-28 00:05:50 UTC) orbiter warning: context deadline exceeded Duration: 3.542862509s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)