Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X....................X...........X..X...........] duration: 26.484280114s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:22:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 26.484296265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X....................X...........X..X..........] duration: 35.44299039s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:22:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 35.443086373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......X....X....X......X....X....X.....XXX..X..XX.......XXX] duration: 30.002692058s trace: https://jaeger.dev.storj.io/trace/1f36590ce20e0ec2 FAIL (2026-01-01 00:22:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: DT3N4LQ7XW2CBIEEHLRAEW67QSRO3PKNZWD6U3CHGBRMRHHGCWNA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: LWQXDNWKTTRCN6V5GCQS56SMHVRUL42MEAI2LVFHCFSPIQA5OX5A) context deadline exceeded ======LOGS====== 1 00:21:33.630848542 (632h52m33.501931577s) [upload=554565]: Starting upload Jan 1 00:21:33.630946107 (632h52m33.502029172s) [upload=554565]: Done starting upload Jan 1 00:21:33.631043953 (632h52m33.502127018s) [upload=554565]: Uploading segments... Jan 1 00:21:33.631359364 (632h52m33.502442409s) [upload=554565]: Got next segment. Inline: false Jan 1 00:21:33.631405412 (632h52m33.502488457s) [upload=554565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:21:33.631411503 (632h52m33.502494548s) [upload=554565]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:21:33.876722262 (632h52m33.747805297s) [seg_pos={0 0}, upload=554565]: Begin upload segment... Jan 1 00:21:33.876763361 (632h52m33.747846396s) [seg_pos={0 0}, upload=554565]: Done begin upload segment. Jan 1 00:21:33.876788228 (632h52m33.747871263s) [noise=true , address=45.91.116.5:4300 , piece_id=DT3N4LQ7XW2CBIEEHLRAEW67QSRO3PKNZWD6U3CHGBRMRHHGCWNA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.876816542 (632h52m33.747899587s) [noise=true , address=145.79.227.19:4078 , piece_id=ERK3BZUBUPL6BJI6VOG7ANMYCA6M7Q3F4XSPHC4RQJXWBINGE5EA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.877252221 (632h52m33.748335256s) [noise=true , address=145.79.227.48:4301 , piece_id=A4ITT5G2DMH7JTRCHBB6OL3TGIEFYD4D2UGGAO6Q5RUXQTL3KQOA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.877634249 (632h52m33.748717284s) [noise=true , address=45.91.116.10:4083 , piece_id=D3O6AMCG5ADSCKZQH3Y47VHURMVAVADMSUV27YMQAS3HQJ2WLLKA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.877969017 (632h52m33.749052062s) [noise=true , address=145.79.227.17:4971 , piece_id=XNMYSD3JSWNRNU76C5LL2IL56D7T5KMMZIACPYCYU6NORNKF67TQ, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.878337498 (632h52m33.749420543s) [noise=true , address=45.91.116.6:4973 , piece_id=WD73R5DNESSGFWC6DPSXWNFW3U7NYXI4D3IGKYKJPQXK2RGHIBGQ, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.878692154 (632h52m33.749775189s) [noise=true , address=77.90.31.13:4626 , piece_id=L5XW3F52EPWNKFZ4CJ4ZE5SG6WPKCANL7B2FN6P5KM2M55BFMF4A, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.879025518 (632h52m33.750108563s) [noise=true , address=77.90.31.17:4770 , piece_id=P5OJ2TCKQ2P2XGLQFLO7YOE3VX6NTRF6ZF74SF2ODOY5OKPLYINA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.879372829 (632h52m33.750455875s) [noise=true , address=145.79.227.46:4602 , piece_id=74GPSADMNEKE73SYOEMHH274OF7YUEKJQVUD3SUDK5CI2HQRFE5A, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.880784026 (632h52m33.751867061s) [noise=true , address=77.90.31.19:4842 , piece_id=LWQXDNWKTTRCN6V5GCQS56SMHVRUL42MEAI2LVFHCFSPIQA5OX5A, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.882003869 (632h52m33.753086904s) [noise=true , address=145.79.227.51:4411 , piece_id=2XEHASGSUE7I7IUWOAAWTNZPBVMTNCAJ33U6F5FAHH6HTGGH52VA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.883363027 (632h52m33.754446072s) [noise=true , address=145.79.227.14:4837 , piece_id=W3EUA3WO2AXNHZV5IR3OCBCL772HBJR34WMXGKEHJCMDP2JOVJDA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.884625782 (632h52m33.755708817s) [noise=true , address=145.79.227.5:4222 , piece_id=T3IWYN2FDODLRIM7JO6WG6FRQC56GHBZPZEG6HBIHWL4N7LA4WOQ, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.886283528 (632h52m33.757366563s) [noise=true , address=145.79.227.20:4438 , piece_id=5UP3CT4HUIF5E7AN7KIBTRUUQZOZIHK77V3FNG74PFMHTCRAIROQ, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.889438566 (632h52m33.760521601s) [noise=true , address=77.90.31.10:4519 , piece_id=I6TJDIULSJBROSYO4LORISMMLUMWTC4ZR3ZAZTRVHP3MEHKAJU7Q, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.890686111 (632h52m33.761769156s) [noise=true , address=45.91.116.9:4263 , piece_id=DB4JDIWLIICNOOFPTEEB6Q2WOHETH47DZU4E6ZCXGWJO6KBZ7ULQ, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:35.590675361 (632h52m35.461758396s) [upload=554565]: Next returned nil segment Jan 1 00:21:35.590700298 (632h52m35.461783363s) [upload=554565]: Waiting for error group managing segments... Jan 1 00:21:35.874771532 (632h52m35.745854567s) [noise=true , address=145.79.227.5:4222 , piece_id=T3IWYN2FDODLRIM7JO6WG6FRQC56GHBZPZEG6HBIHWL4N7LA4WOQ, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.881318347 (632h52m35.752401382s) [noise=true , address=145.79.227.46:4602 , piece_id=74GPSADMNEKE73SYOEMHH274OF7YUEKJQVUD3SUDK5CI2HQRFE5A, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.881965350 (632h52m35.753048385s) [noise=true , address=145.79.227.51:4411 , piece_id=2XEHASGSUE7I7IUWOAAWTNZPBVMTNCAJ33U6F5FAHH6HTGGH52VA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.886008498 (632h52m35.757091543s) [noise=true , address=145.79.227.19:4078 , piece_id=ERK3BZUBUPL6BJI6VOG7ANMYCA6M7Q3F4XSPHC4RQJXWBINGE5EA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.890818297 (632h52m35.761901352s) [noise=true , address=145.79.227.48:4301 , piece_id=A4ITT5G2DMH7JTRCHBB6OL3TGIEFYD4D2UGGAO6Q5RUXQTL3KQOA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.895023022 (632h52m35.766106068s) [noise=true , address=145.79.227.20:4438 , piece_id=5UP3CT4HUIF5E7AN7KIBTRUUQZOZIHK77V3FNG74PFMHTCRAIROQ, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.901113640 (632h52m35.772196685s) [noise=true , address=145.79.227.14:4837 , piece_id=W3EUA3WO2AXNHZV5IR3OCBCL772HBJR34WMXGKEHJCMDP2JOVJDA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.032899505 (632h52m35.90398255s) [noise=true , address=77.90.31.13:4626 , piece_id=L5XW3F52EPWNKFZ4CJ4ZE5SG6WPKCANL7B2FN6P5KM2M55BFMF4A, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.113508915 (632h52m35.98459195s) [noise=true , address=77.90.31.10:4519 , piece_id=I6TJDIULSJBROSYO4LORISMMLUMWTC4ZR3ZAZTRVHP3MEHKAJU7Q, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.156699655 (632h52m36.02778269s) [noise=true , address=145.79.227.17:4971 , piece_id=XNMYSD3JSWNRNU76C5LL2IL56D7T5KMMZIACPYCYU6NORNKF67TQ, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.192463022 (632h52m36.063546067s) [noise=true , address=45.91.116.6:4973 , piece_id=WD73R5DNESSGFWC6DPSXWNFW3U7NYXI4D3IGKYKJPQXK2RGHIBGQ, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.284587938 (632h52m36.155670974s) [noise=true , address=45.91.116.5:4300 , piece_id=DT3N4LQ7XW2CBIEEHLRAEW67QSRO3PKNZWD6U3CHGBRMRHHGCWNA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.820064066 (632h52m36.691147101s) [noise=true , address=77.90.31.19:4842 , piece_id=LWQXDNWKTTRCN6V5GCQS56SMHVRUL42MEAI2LVFHCFSPIQA5OX5A, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.820127145 (632h52m36.69121019s) [seg_pos={0 0}, upload=554565]: Segment reached optimal threshold of 13 pieces. Jan 1 00:21:36.820294364 (632h52m36.691377399s) [noise=true , address=45.91.116.9:4263 , piece_id=DB4JDIWLIICNOOFPTEEB6Q2WOHETH47DZU4E6ZCXGWJO6KBZ7ULQ, seg_pos={0 0}, upload=554565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:45112->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:45112->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45112->45.91.116.9:4263: use of closed network connection Jan 1 00:21:36.820300636 (632h52m36.691383671s) [noise=true , address=45.91.116.9:4263 , piece_id=DB4JDIWLIICNOOFPTEEB6Q2WOHETH47DZU4E6ZCXGWJO6KBZ7ULQ, seg_pos={0 0}, upload=554565]: Long tail cancellation due to optimal threshold Jan 1 00:21:36.820374376 (632h52m36.691457421s) [noise=true , address=77.90.31.17:4770 , piece_id=P5OJ2TCKQ2P2XGLQFLO7YOE3VX6NTRF6ZF74SF2ODOY5OKPLYINA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 1 00:21:36.820404023 (632h52m36.691487058s) [noise=true , address=77.90.31.17:4770 , piece_id=P5OJ2TCKQ2P2XGLQFLO7YOE3VX6NTRF6ZF74SF2ODOY5OKPLYINA, seg_pos={0 0}, upload=554565]: Long tail cancellation due to optimal threshold Jan 1 00:21:36.820519924 (632h52m36.691602959s) [noise=true , address=45.91.116.10:4083 , piece_id=D3O6AMCG5ADSCKZQH3Y47VHURMVAVADMSUV27YMQAS3HQJ2WLLKA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 1 00:21:36.820526576 (632h52m36.691609611s) [noise=true , address=45.91.116.10:4083 , piece_id=D3O6AMCG5ADSCKZQH3Y47VHURMVAVADMSUV27YMQAS3HQJ2WLLKA, seg_pos={0 0}, upload=554565]: Long tail cancellation due to optimal threshold Jan 1 00:21:36.820542326 (632h52m36.691625371s) [seg_pos={0 0}, upload=554565]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:21:36.820607059 (632h52m36.691690094s) [upload=554565]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:21:36.820611828 (632h52m36.691694883s) [upload=554565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:21:37.129381360 (632h52m37.000464395s) [upload=554565]: Done uploading segments... Jan 1 00:21:37.129402531 (632h52m37.000485566s) [upload=554565]: Upload finished. err: <nil> Duration: 30.00271907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........X....X....X......X....X....X.....XXX..X..XX.......XX] duration: 30.003160152s trace: https://jaeger.dev.storj.io/trace/4f76ac31dd4a689a FAIL (2026-01-01 00:21:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 00:21:00.439449313 (632h52m0.310532348s) [upload=554559]: Starting upload Jan 1 00:21:00.439509809 (632h52m0.310592854s) [upload=554559]: Done starting upload Jan 1 00:21:00.439599529 (632h52m0.310682564s) [upload=554559]: Uploading segments... Jan 1 00:21:00.439697315 (632h52m0.31078035s) [upload=554559]: Got next segment. Inline: false Jan 1 00:21:00.439733825 (632h52m0.31081686s) [upload=554559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:21:00.439738293 (632h52m0.310821328s) [upload=554559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:21:00.689726850 (632h52m0.560809885s) [seg_pos={0 0}, upload=554559]: Begin upload segment... Jan 1 00:21:00.689775953 (632h52m0.560858988s) [seg_pos={0 0}, upload=554559]: Done begin upload segment. Jan 1 00:21:00.689836920 (632h52m0.560919955s) [noise=true , address=45.91.116.7:4012 , piece_id=2NRIZ6ZSRAHJTKFFRASP5NGJODACHBO77VZKYFLPQFWUBAVL3W2Q, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.689954383 (632h52m0.561037428s) [noise=true , address=145.79.227.18:4006 , piece_id=N6KNUCPTXLHDM4JRJH3M5NCMBCB3NBXUYLENFYGK3E6MJVWNCPWA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.690497097 (632h52m0.561580142s) [noise=true , address=145.79.227.10:4403 , piece_id=RFOMOAVHECHJRI7SACAD5T3MSKVTSKQIFIB67MREPM24GFYKBNOA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.691003051 (632h52m0.562086096s) [noise=true , address=145.79.227.46:4601 , piece_id=74ZPHMXL23MP7PLEYGXQEAYL6AOP7Q67YYCSFEYK2QXBRPQQZTYQ, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.691059177 (632h52m0.562142222s) [noise=true , address=145.79.227.27:4690 , piece_id=TX4NTRDLW4ZXVGXIU34MPRSX2AFMKI3JX3AFTGBUS6ZBF7MM2HXA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.691434041 (632h52m0.562517076s) [noise=true , address=77.90.31.15:4700 , piece_id=B7MZROSA6W65OITURSTCM7HYYX2A5T3AECTIQVQK5YQZ2FXD44YQ, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.691754000 (632h52m0.562837035s) [noise=true , address=145.79.227.7:4293 , piece_id=XL4BBMISRRZKIEHXKZNBIEPFNEB2ANDGJEF2HGDGSZTZAAKRAJCA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.691778888 (632h52m0.562861933s) [noise=true , address=145.79.227.12:4767 , piece_id=LPRJFKZM3G42VAKHJ6G7OA5WRASSKRNIWMQJE2EJSYT4OKOFLJEA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.692066946 (632h52m0.563149981s) [noise=true , address=145.79.227.17:4971 , piece_id=HSUKY3YJZVHD2STH7RZPDFTN5HFQAWFJNRWK3VJX3EN7NZZXN5XQ, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.692436500 (632h52m0.563519575s) [noise=true , address=77.90.31.17:4770 , piece_id=TF6UASJBBHPQLXZ5ULGLJ5E2BJKJY2U4MUVMB5XU3JJCXP5LZZ6Q, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.692458662 (632h52m0.563541697s) [noise=true , address=45.91.116.4:4940 , piece_id=ZCWQBZLIJ2OUH6QUI4MDLAIU4KJ2QAHHMGXLXTINLEAKVMTX6Q7A, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.692867280 (632h52m0.563950315s) [noise=true , address=77.90.31.13:4626 , piece_id=WFZIADBXP2YA5SSKOX4AL32AUS3W7SXUDROO62A4CKYXUE66HTYA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.693184053 (632h52m0.564267098s) [noise=true , address=145.79.227.41:4087 , piece_id=TZL2YWVZDPSJIWRORLH2NIIWGP4B7SWT75ETAOKWJIY7E6H2VXHQ, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.694008924 (632h52m0.565091959s) [noise=true , address=145.79.227.34:4905 , piece_id=TPSGJS73IS3JRHTL5KKIJKC5ULTY4Q47IKHMUICIJ44IPYTFVAQA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.694451718 (632h52m0.565534753s) [noise=true , address=145.79.227.5:4221 , piece_id=7J2ONDDC4MV25Y7MLWDNOSN3T25L5CNMY2C6UB6UQPN5CHXQCO6A, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.694778459 (632h52m0.565861494s) [noise=true , address=77.90.31.10:4520 , piece_id=RQWOIAD6MAKD3Y2FUSQYJX2BL2UAOYO4Z5HERZP6XX4WRWDBFMMA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:02.385856803 (632h52m2.256939848s) [upload=554559]: Next returned nil segment Jan 1 00:21:02.385880117 (632h52m2.256963162s) [upload=554559]: Waiting for error group managing segments... Jan 1 00:21:02.669350838 (632h52m2.540433873s) [noise=true , address=145.79.227.10:4403 , piece_id=RFOMOAVHECHJRI7SACAD5T3MSKVTSKQIFIB67MREPM24GFYKBNOA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.678588578 (632h52m2.549671623s) [noise=true , address=145.79.227.18:4006 , piece_id=N6KNUCPTXLHDM4JRJH3M5NCMBCB3NBXUYLENFYGK3E6MJVWNCPWA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.683946510 (632h52m2.555029555s) [noise=true , address=145.79.227.46:4601 , piece_id=74ZPHMXL23MP7PLEYGXQEAYL6AOP7Q67YYCSFEYK2QXBRPQQZTYQ, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.685121086 (632h52m2.556204131s) [noise=true , address=145.79.227.27:4690 , piece_id=TX4NTRDLW4ZXVGXIU34MPRSX2AFMKI3JX3AFTGBUS6ZBF7MM2HXA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.690715128 (632h52m2.561798163s) [noise=true , address=145.79.227.17:4971 , piece_id=HSUKY3YJZVHD2STH7RZPDFTN5HFQAWFJNRWK3VJX3EN7NZZXN5XQ, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.698269993 (632h52m2.569353039s) [noise=true , address=145.79.227.5:4221 , piece_id=7J2ONDDC4MV25Y7MLWDNOSN3T25L5CNMY2C6UB6UQPN5CHXQCO6A, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.702874671 (632h52m2.573957716s) [noise=true , address=145.79.227.41:4087 , piece_id=TZL2YWVZDPSJIWRORLH2NIIWGP4B7SWT75ETAOKWJIY7E6H2VXHQ, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.711320864 (632h52m2.582403899s) [noise=true , address=145.79.227.12:4767 , piece_id=LPRJFKZM3G42VAKHJ6G7OA5WRASSKRNIWMQJE2EJSYT4OKOFLJEA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.846944697 (632h52m2.718027742s) [noise=true , address=77.90.31.13:4626 , piece_id=WFZIADBXP2YA5SSKOX4AL32AUS3W7SXUDROO62A4CKYXUE66HTYA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.967381441 (632h52m2.838464487s) [noise=true , address=77.90.31.10:4520 , piece_id=RQWOIAD6MAKD3Y2FUSQYJX2BL2UAOYO4Z5HERZP6XX4WRWDBFMMA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.977755275 (632h52m2.84883831s) [noise=true , address=45.91.116.4:4940 , piece_id=ZCWQBZLIJ2OUH6QUI4MDLAIU4KJ2QAHHMGXLXTINLEAKVMTX6Q7A, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.989714928 (632h52m2.860797973s) [noise=true , address=45.91.116.7:4012 , piece_id=2NRIZ6ZSRAHJTKFFRASP5NGJODACHBO77VZKYFLPQFWUBAVL3W2Q, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:03.573856028 (632h52m3.444939073s) [noise=true , address=77.90.31.17:4770 , piece_id=TF6UASJBBHPQLXZ5ULGLJ5E2BJKJY2U4MUVMB5XU3JJCXP5LZZ6Q, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:03.573885193 (632h52m3.444968239s) [seg_pos={0 0}, upload=554559]: Segment reached optimal threshold of 13 pieces. Jan 1 00:21:03.574217155 (632h52m3.44530021s) [noise=true , address=145.79.227.7:4293 , piece_id=XL4BBMISRRZKIEHXKZNBIEPFNEB2ANDGJEF2HGDGSZTZAAKRAJCA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:40334->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:40334->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40334->145.79.227.7:4293: use of closed network connection Jan 1 00:21:03.574228127 (632h52m3.445311172s) [noise=true , address=145.79.227.7:4293 , piece_id=XL4BBMISRRZKIEHXKZNBIEPFNEB2ANDGJEF2HGDGSZTZAAKRAJCA, seg_pos={0 0}, upload=554559]: Long tail cancellation due to optimal threshold Jan 1 00:21:03.574233948 (632h52m3.445316993s) [noise=true , address=77.90.31.15:4700 , piece_id=B7MZROSA6W65OITURSTCM7HYYX2A5T3AECTIQVQK5YQZ2FXD44YQ, seg_pos={0 0}, upload=554559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Jan 1 00:21:03.574250659 (632h52m3.445333694s) [noise=true , address=77.90.31.15:4700 , piece_id=B7MZROSA6W65OITURSTCM7HYYX2A5T3AECTIQVQK5YQZ2FXD44YQ, seg_pos={0 0}, upload=554559]: Long tail cancellation due to optimal threshold Jan 1 00:21:03.574489824 (632h52m3.445572869s) [noise=true , address=145.79.227.34:4905 , piece_id=TPSGJS73IS3JRHTL5KKIJKC5ULTY4Q47IKHMUICIJ44IPYTFVAQA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 1 00:21:03.574497619 (632h52m3.445580664s) [noise=true , address=145.79.227.34:4905 , piece_id=TPSGJS73IS3JRHTL5KKIJKC5ULTY4Q47IKHMUICIJ44IPYTFVAQA, seg_pos={0 0}, upload=554559]: Long tail cancellation due to optimal threshold Jan 1 00:21:03.574684335 (632h52m3.44576738s) [seg_pos={0 0}, upload=554559]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:21:03.574809633 (632h52m3.445892678s) [upload=554559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:21:03.574817318 (632h52m3.445900363s) [upload=554559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:21:03.875621995 (632h52m3.74670504s) [upload=554559]: Done uploading segments... Jan 1 00:21:03.875643725 (632h52m3.74672677s) [upload=554559]: Upload finished. err: <nil> Duration: 30.00318086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X....................X...........X..X.........] duration: 21.446036821s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:21:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 21.446053715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X....................X...........X..X........] duration: 32.166741072s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:20:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 32.166771271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........X....X....X......X....X....X.....XXX..X..XX.......X] duration: 30.002553978s trace: https://jaeger.dev.storj.io/trace/42734d11284293f4 FAIL (2026-01-01 00:20:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: AJ4JM3GIVUV6SN6H7DO57EWC3DNUMSJZVXLHPWU2VLVMISZMIHBA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: QULGJQGLCDRPT3PHD3IYNWMD5EVIUJN7NCGYQK2KPDZEEGPMXRAA) context deadline exceeded ======LOGS====== 0734s) [upload=554545]: Uploading segments... Jan 1 00:20:07.677148691 (632h51m7.548231726s) [upload=554545]: Got next segment. Inline: false Jan 1 00:20:07.677198635 (632h51m7.54828167s) [upload=554545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:20:07.677207251 (632h51m7.548290327s) [upload=554545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:20:07.926271698 (632h51m7.797354743s) [seg_pos={0 0}, upload=554545]: Begin upload segment... Jan 1 00:20:07.926347072 (632h51m7.797430117s) [seg_pos={0 0}, upload=554545]: Done begin upload segment. Jan 1 00:20:07.926397147 (632h51m7.797480192s) [noise=true , address=145.79.227.8:4329 , piece_id=FGAFTS5L3WCYHM4C6MDBR6ECS4Z3MIYIMALX6ZPDJJ3XSGQEEEPQ, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.926404420 (632h51m7.797487455s) [noise=true , address=145.79.227.50:4374 , piece_id=UTLQPODPZVEBH7KTKIB5PJVCMKLVGKMNRDBKJVBUDUBVMOYUAH5Q, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.927633431 (632h51m7.798716476s) [noise=true , address=45.91.116.10:4084 , piece_id=7HDGWCORGC2IRAHUDQBBG4R5KEYWYHTXSJV4GZSWZT7HDOZDYTFQ, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.928195832 (632h51m7.799278877s) [noise=true , address=145.79.227.41:4087 , piece_id=OGNUJMTC2MXSCAWU3SMULUWKBGRGFNBZBNGTPPH34DYMG6O6H3FA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.928698709 (632h51m7.799781754s) [noise=true , address=145.79.227.10:4403 , piece_id=OTME4XMJCXS4YI6DEBZEEFD76VAG5BGYB4FSBGAGFW4PYZD72QQA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.929177762 (632h51m7.800260797s) [noise=true , address=45.91.116.13:4153 , piece_id=DDNJLAZDZXG4BCG5YAEIUSYNV5XXKYGLJXUQTEDOOEJ2FRNRYWKA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.929188872 (632h51m7.800271917s) [noise=true , address=145.79.227.42:4121 , piece_id=VBZRP5HSKD3ILSGOQXLJTVMMAJFZSWSKDDYZXLKFPTI3J3BQ5RMA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.929639130 (632h51m7.800722175s) [noise=true , address=145.79.227.12:4765 , piece_id=IV7TL4SHQIVM6VXMCIELJ7ALODEY2YTW4JTW7PQRKWOD27ARDZZA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.929794837 (632h51m7.800877882s) [noise=true , address=45.91.116.5:4298 , piece_id=T6LLFL2PYC6VWFKFKRBEDGJZ2EAHPQ3B4RIFVUF3GJETYEJENEOA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.930213534 (632h51m7.801296579s) [noise=true , address=77.90.31.13:4627 , piece_id=AJ4JM3GIVUV6SN6H7DO57EWC3DNUMSJZVXLHPWU2VLVMISZMIHBA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.930740517 (632h51m7.801823552s) [noise=true , address=77.90.31.11:4554 , piece_id=KGMH7SSYWOZLI2ELTSQFYIS2SDJMLKDNVH4AFR5DCQLBMMEDTB6A, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.930875825 (632h51m7.80195887s) [noise=true , address=45.91.116.4:4939 , piece_id=LQ7XGBX6KKIKDHQZHVTKP7CKUY3AYJFMG7UMEVSRSTHK3DGSFM2Q, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.931250198 (632h51m7.802333253s) [noise=true , address=145.79.227.4:4186 , piece_id=JYN7LDBIMYGUXWALY7VTABQI6ZZ5GQXGJ4DHDZIR7XN3Y2HYN6KQ, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.931784235 (632h51m7.80286727s) [noise=true , address=45.91.116.11:4119 , piece_id=XIU2UAKGRIIK7CW7LLNWKXZFHY3SGFZ7DZ7BY4QVS7KP64UK5FNQ, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.931998944 (632h51m7.803081989s) [noise=true , address=77.90.31.21:4877 , piece_id=QULGJQGLCDRPT3PHD3IYNWMD5EVIUJN7NCGYQK2KPDZEEGPMXRAA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.932303103 (632h51m7.803386158s) [noise=true , address=145.79.227.11:4732 , piece_id=QGVVSC6DU3MYZYU3HC6ZBG4YQDND7PISGNDC7PMQRKFTW6FOBZ4Q, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:09.640073018 (632h51m9.511156053s) [upload=554545]: Next returned nil segment Jan 1 00:20:09.640091534 (632h51m9.511174579s) [upload=554545]: Waiting for error group managing segments... Jan 1 00:20:09.929463480 (632h51m9.800546525s) [noise=true , address=145.79.227.50:4374 , piece_id=UTLQPODPZVEBH7KTKIB5PJVCMKLVGKMNRDBKJVBUDUBVMOYUAH5Q, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:09.935030982 (632h51m9.806114027s) [noise=true , address=145.79.227.41:4087 , piece_id=OGNUJMTC2MXSCAWU3SMULUWKBGRGFNBZBNGTPPH34DYMG6O6H3FA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:09.942689664 (632h51m9.813772709s) [noise=true , address=145.79.227.4:4186 , piece_id=JYN7LDBIMYGUXWALY7VTABQI6ZZ5GQXGJ4DHDZIR7XN3Y2HYN6KQ, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:09.944484212 (632h51m9.815567257s) [noise=true , address=145.79.227.11:4732 , piece_id=QGVVSC6DU3MYZYU3HC6ZBG4YQDND7PISGNDC7PMQRKFTW6FOBZ4Q, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:09.944720061 (632h51m9.815803106s) [noise=true , address=145.79.227.8:4329 , piece_id=FGAFTS5L3WCYHM4C6MDBR6ECS4Z3MIYIMALX6ZPDJJ3XSGQEEEPQ, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:09.980720801 (632h51m9.851803846s) [noise=true , address=145.79.227.12:4765 , piece_id=IV7TL4SHQIVM6VXMCIELJ7ALODEY2YTW4JTW7PQRKWOD27ARDZZA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.001444972 (632h51m9.872528027s) [noise=true , address=145.79.227.42:4121 , piece_id=VBZRP5HSKD3ILSGOQXLJTVMMAJFZSWSKDDYZXLKFPTI3J3BQ5RMA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.006359599 (632h51m9.877442634s) [noise=true , address=145.79.227.10:4403 , piece_id=OTME4XMJCXS4YI6DEBZEEFD76VAG5BGYB4FSBGAGFW4PYZD72QQA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.120073393 (632h51m9.991156428s) [noise=true , address=77.90.31.13:4627 , piece_id=AJ4JM3GIVUV6SN6H7DO57EWC3DNUMSJZVXLHPWU2VLVMISZMIHBA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.212869397 (632h51m10.083952462s) [noise=true , address=45.91.116.11:4119 , piece_id=XIU2UAKGRIIK7CW7LLNWKXZFHY3SGFZ7DZ7BY4QVS7KP64UK5FNQ, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.221826693 (632h51m10.092909738s) [noise=true , address=77.90.31.21:4877 , piece_id=QULGJQGLCDRPT3PHD3IYNWMD5EVIUJN7NCGYQK2KPDZEEGPMXRAA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.264804738 (632h51m10.135887783s) [noise=true , address=45.91.116.5:4298 , piece_id=T6LLFL2PYC6VWFKFKRBEDGJZ2EAHPQ3B4RIFVUF3GJETYEJENEOA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.344870752 (632h51m10.215953797s) [noise=true , address=45.91.116.10:4084 , piece_id=7HDGWCORGC2IRAHUDQBBG4R5KEYWYHTXSJV4GZSWZT7HDOZDYTFQ, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.344894267 (632h51m10.215977312s) [seg_pos={0 0}, upload=554545]: Segment reached optimal threshold of 13 pieces. Jan 1 00:20:10.345081023 (632h51m10.216164058s) [noise=true , address=77.90.31.11:4554 , piece_id=KGMH7SSYWOZLI2ELTSQFYIS2SDJMLKDNVH4AFR5DCQLBMMEDTB6A, seg_pos={0 0}, upload=554545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:60870->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:60870->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->77.90.31.11:4554: use of closed network connection Jan 1 00:20:10.345087585 (632h51m10.21617063s) [noise=true , address=77.90.31.11:4554 , piece_id=KGMH7SSYWOZLI2ELTSQFYIS2SDJMLKDNVH4AFR5DCQLBMMEDTB6A, seg_pos={0 0}, upload=554545]: Long tail cancellation due to optimal threshold Jan 1 00:20:10.345149904 (632h51m10.216232949s) [noise=true , address=45.91.116.4:4939 , piece_id=LQ7XGBX6KKIKDHQZHVTKP7CKUY3AYJFMG7UMEVSRSTHK3DGSFM2Q, seg_pos={0 0}, upload=554545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:33590->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:33590->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33590->45.91.116.4:4939: use of closed network connection Jan 1 00:20:10.345159221 (632h51m10.216242256s) [noise=true , address=45.91.116.4:4939 , piece_id=LQ7XGBX6KKIKDHQZHVTKP7CKUY3AYJFMG7UMEVSRSTHK3DGSFM2Q, seg_pos={0 0}, upload=554545]: Long tail cancellation due to optimal threshold Jan 1 00:20:10.345325428 (632h51m10.216408473s) [noise=true , address=45.91.116.13:4153 , piece_id=DDNJLAZDZXG4BCG5YAEIUSYNV5XXKYGLJXUQTEDOOEJ2FRNRYWKA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Jan 1 00:20:10.345335156 (632h51m10.216418201s) [noise=true , address=45.91.116.13:4153 , piece_id=DDNJLAZDZXG4BCG5YAEIUSYNV5XXKYGLJXUQTEDOOEJ2FRNRYWKA, seg_pos={0 0}, upload=554545]: Long tail cancellation due to optimal threshold Jan 1 00:20:10.345356627 (632h51m10.216439663s) [seg_pos={0 0}, upload=554545]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:20:10.345448493 (632h51m10.216531528s) [upload=554545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:20:10.345454825 (632h51m10.21653787s) [upload=554545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:20:10.642646802 (632h51m10.513729837s) [upload=554545]: Done uploading segments... Jan 1 00:20:10.642676177 (632h51m10.513759222s) [upload=554545]: Upload finished. err: <nil> Duration: 30.00257517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X....................X...........X..X.......] duration: 15.302646472s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:19:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 15.302741964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X....X....X......X....X....X.....XXX..X..XX.......] duration: 6.272137389s trace: https://jaeger.dev.storj.io/trace/28a9125ed5f2f5b5 FAIL (2026-01-01 00:19:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Jan 1 00:13:49.620961485 (632h44m49.49204452s) [upload=554454]: Got next segment. Inline: false Jan 1 00:13:49.620997573 (632h44m49.492080618s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:13:49.621002192 (632h44m49.492085237s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:13:49.868640814 (632h44m49.739723849s) [seg_pos={0 0}, upload=554454]: Begin upload segment... Jan 1 00:13:49.868705798 (632h44m49.739788833s) [seg_pos={0 0}, upload=554454]: Done begin upload segment. Jan 1 00:13:49.868724784 (632h44m49.739807819s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868748168 (632h44m49.739831203s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868826719 (632h44m49.739909764s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869134614 (632h44m49.740217659s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869399539 (632h44m49.740482574s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870161510 (632h44m49.741244545s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870452644 (632h44m49.741535679s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870806107 (632h44m49.741889152s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871111188 (632h44m49.742194223s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871466394 (632h44m49.742549429s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872254716 (632h44m49.743337751s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872531743 (632h44m49.743614788s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.873624453 (632h44m49.744707489s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.874767320 (632h44m49.745850355s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.875797311 (632h44m49.746880346s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.876955366 (632h44m49.748038401s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:51.575471743 (632h44m51.446554788s) [upload=554454]: Next returned nil segment Jan 1 00:13:51.575676152 (632h44m51.446759227s) [upload=554454]: Waiting for error group managing segments... Jan 1 00:13:51.855996814 (632h44m51.727079849s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.864465159 (632h44m51.735548204s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.868352721 (632h44m51.739435756s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.881984337 (632h44m51.753067382s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.884508242 (632h44m51.755591277s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.896797463 (632h44m51.767880498s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.942212709 (632h44m51.813295744s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.947553388 (632h44m51.818636443s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.017191837 (632h44m51.888274882s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.157166414 (632h44m52.028249469s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.166689146 (632h44m52.037772192s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.240675014 (632h44m52.111758059s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301683050 (632h44m52.172766095s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301707176 (632h44m52.172790222s) [seg_pos={0 0}, upload=554454]: Segment reached optimal threshold of 13 pieces. Jan 1 00:13:52.301980036 (632h44m52.173063081s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:52.301991337 (632h44m52.173074372s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302201427 (632h44m52.173284472s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection Jan 1 00:13:52.302211015 (632h44m52.17329406s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302356243 (632h44m52.173439278s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection Jan 1 00:13:52.302366773 (632h44m52.173449808s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302390067 (632h44m52.173473102s) [seg_pos={0 0}, upload=554454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:13:52.302479958 (632h44m52.173563003s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:52.302487141 (632h44m52.173570186s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:52.601605125 (632h44m52.47268817s) [upload=554454]: Done uploading segments... Jan 1 00:13:52.601627638 (632h44m52.472710673s) [upload=554454]: Upload finished. err: <nil> Duration: 6.272162858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X....................X...........X..X......] duration: 28.727550213s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:18:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 28.727574591s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........X....X....X......X....X....X.....XXX..X..XX......] duration: 5.911875577s trace: https://jaeger.dev.storj.io/trace/28a9125ed5f2f5b5 FAIL (2026-01-01 00:18:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Jan 1 00:13:49.620961485 (632h44m49.49204452s) [upload=554454]: Got next segment. Inline: false Jan 1 00:13:49.620997573 (632h44m49.492080618s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:13:49.621002192 (632h44m49.492085237s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:13:49.868640814 (632h44m49.739723849s) [seg_pos={0 0}, upload=554454]: Begin upload segment... Jan 1 00:13:49.868705798 (632h44m49.739788833s) [seg_pos={0 0}, upload=554454]: Done begin upload segment. Jan 1 00:13:49.868724784 (632h44m49.739807819s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868748168 (632h44m49.739831203s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868826719 (632h44m49.739909764s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869134614 (632h44m49.740217659s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869399539 (632h44m49.740482574s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870161510 (632h44m49.741244545s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870452644 (632h44m49.741535679s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870806107 (632h44m49.741889152s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871111188 (632h44m49.742194223s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871466394 (632h44m49.742549429s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872254716 (632h44m49.743337751s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872531743 (632h44m49.743614788s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.873624453 (632h44m49.744707489s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.874767320 (632h44m49.745850355s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.875797311 (632h44m49.746880346s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.876955366 (632h44m49.748038401s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:51.575471743 (632h44m51.446554788s) [upload=554454]: Next returned nil segment Jan 1 00:13:51.575676152 (632h44m51.446759227s) [upload=554454]: Waiting for error group managing segments... Jan 1 00:13:51.855996814 (632h44m51.727079849s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.864465159 (632h44m51.735548204s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.868352721 (632h44m51.739435756s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.881984337 (632h44m51.753067382s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.884508242 (632h44m51.755591277s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.896797463 (632h44m51.767880498s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.942212709 (632h44m51.813295744s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.947553388 (632h44m51.818636443s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.017191837 (632h44m51.888274882s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.157166414 (632h44m52.028249469s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.166689146 (632h44m52.037772192s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.240675014 (632h44m52.111758059s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301683050 (632h44m52.172766095s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301707176 (632h44m52.172790222s) [seg_pos={0 0}, upload=554454]: Segment reached optimal threshold of 13 pieces. Jan 1 00:13:52.301980036 (632h44m52.173063081s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:52.301991337 (632h44m52.173074372s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302201427 (632h44m52.173284472s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection Jan 1 00:13:52.302211015 (632h44m52.17329406s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302356243 (632h44m52.173439278s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection Jan 1 00:13:52.302366773 (632h44m52.173449808s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302390067 (632h44m52.173473102s) [seg_pos={0 0}, upload=554454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:13:52.302479958 (632h44m52.173563003s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:52.302487141 (632h44m52.173570186s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:52.601605125 (632h44m52.47268817s) [upload=554454]: Done uploading segments... Jan 1 00:13:52.601627638 (632h44m52.472710673s) [upload=554454]: Upload finished. err: <nil> Duration: 5.911901557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X....................X...........X..X.....] duration: 34.615710396s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:17:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 34.615734211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............X....X....X......X....X....X.....XXX..X..XX.....] duration: 6.007667294s trace: https://jaeger.dev.storj.io/trace/28a9125ed5f2f5b5 FAIL (2026-01-01 00:17:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Jan 1 00:13:49.620961485 (632h44m49.49204452s) [upload=554454]: Got next segment. Inline: false Jan 1 00:13:49.620997573 (632h44m49.492080618s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:13:49.621002192 (632h44m49.492085237s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:13:49.868640814 (632h44m49.739723849s) [seg_pos={0 0}, upload=554454]: Begin upload segment... Jan 1 00:13:49.868705798 (632h44m49.739788833s) [seg_pos={0 0}, upload=554454]: Done begin upload segment. Jan 1 00:13:49.868724784 (632h44m49.739807819s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868748168 (632h44m49.739831203s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868826719 (632h44m49.739909764s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869134614 (632h44m49.740217659s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869399539 (632h44m49.740482574s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870161510 (632h44m49.741244545s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870452644 (632h44m49.741535679s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870806107 (632h44m49.741889152s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871111188 (632h44m49.742194223s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871466394 (632h44m49.742549429s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872254716 (632h44m49.743337751s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872531743 (632h44m49.743614788s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.873624453 (632h44m49.744707489s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.874767320 (632h44m49.745850355s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.875797311 (632h44m49.746880346s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.876955366 (632h44m49.748038401s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:51.575471743 (632h44m51.446554788s) [upload=554454]: Next returned nil segment Jan 1 00:13:51.575676152 (632h44m51.446759227s) [upload=554454]: Waiting for error group managing segments... Jan 1 00:13:51.855996814 (632h44m51.727079849s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.864465159 (632h44m51.735548204s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.868352721 (632h44m51.739435756s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.881984337 (632h44m51.753067382s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.884508242 (632h44m51.755591277s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.896797463 (632h44m51.767880498s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.942212709 (632h44m51.813295744s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.947553388 (632h44m51.818636443s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.017191837 (632h44m51.888274882s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.157166414 (632h44m52.028249469s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.166689146 (632h44m52.037772192s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.240675014 (632h44m52.111758059s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301683050 (632h44m52.172766095s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301707176 (632h44m52.172790222s) [seg_pos={0 0}, upload=554454]: Segment reached optimal threshold of 13 pieces. Jan 1 00:13:52.301980036 (632h44m52.173063081s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:52.301991337 (632h44m52.173074372s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302201427 (632h44m52.173284472s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection Jan 1 00:13:52.302211015 (632h44m52.17329406s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302356243 (632h44m52.173439278s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection Jan 1 00:13:52.302366773 (632h44m52.173449808s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302390067 (632h44m52.173473102s) [seg_pos={0 0}, upload=554454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:13:52.302479958 (632h44m52.173563003s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:52.302487141 (632h44m52.173570186s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:52.601605125 (632h44m52.47268817s) [upload=554454]: Done uploading segments... Jan 1 00:13:52.601627638 (632h44m52.472710673s) [upload=554454]: Upload finished. err: <nil> Duration: 6.007692491s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X....................X...........X..X....] duration: 16.570670166s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:17:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 16.570685394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X....................X...........X..X...] duration: 37.067836718s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:17:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 37.067862558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X............X....X....X......X....X....X.....XXX..X..XX....] duration: 5.975345078s trace: https://jaeger.dev.storj.io/trace/28a9125ed5f2f5b5 FAIL (2026-01-01 00:16:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Jan 1 00:13:49.620961485 (632h44m49.49204452s) [upload=554454]: Got next segment. Inline: false Jan 1 00:13:49.620997573 (632h44m49.492080618s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:13:49.621002192 (632h44m49.492085237s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:13:49.868640814 (632h44m49.739723849s) [seg_pos={0 0}, upload=554454]: Begin upload segment... Jan 1 00:13:49.868705798 (632h44m49.739788833s) [seg_pos={0 0}, upload=554454]: Done begin upload segment. Jan 1 00:13:49.868724784 (632h44m49.739807819s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868748168 (632h44m49.739831203s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868826719 (632h44m49.739909764s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869134614 (632h44m49.740217659s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869399539 (632h44m49.740482574s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870161510 (632h44m49.741244545s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870452644 (632h44m49.741535679s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870806107 (632h44m49.741889152s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871111188 (632h44m49.742194223s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871466394 (632h44m49.742549429s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872254716 (632h44m49.743337751s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872531743 (632h44m49.743614788s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.873624453 (632h44m49.744707489s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.874767320 (632h44m49.745850355s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.875797311 (632h44m49.746880346s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.876955366 (632h44m49.748038401s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:51.575471743 (632h44m51.446554788s) [upload=554454]: Next returned nil segment Jan 1 00:13:51.575676152 (632h44m51.446759227s) [upload=554454]: Waiting for error group managing segments... Jan 1 00:13:51.855996814 (632h44m51.727079849s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.864465159 (632h44m51.735548204s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.868352721 (632h44m51.739435756s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.881984337 (632h44m51.753067382s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.884508242 (632h44m51.755591277s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.896797463 (632h44m51.767880498s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.942212709 (632h44m51.813295744s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.947553388 (632h44m51.818636443s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.017191837 (632h44m51.888274882s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.157166414 (632h44m52.028249469s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.166689146 (632h44m52.037772192s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.240675014 (632h44m52.111758059s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301683050 (632h44m52.172766095s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301707176 (632h44m52.172790222s) [seg_pos={0 0}, upload=554454]: Segment reached optimal threshold of 13 pieces. Jan 1 00:13:52.301980036 (632h44m52.173063081s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:52.301991337 (632h44m52.173074372s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302201427 (632h44m52.173284472s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection Jan 1 00:13:52.302211015 (632h44m52.17329406s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302356243 (632h44m52.173439278s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection Jan 1 00:13:52.302366773 (632h44m52.173449808s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302390067 (632h44m52.173473102s) [seg_pos={0 0}, upload=554454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:13:52.302479958 (632h44m52.173563003s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:52.302487141 (632h44m52.173570186s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:52.601605125 (632h44m52.47268817s) [upload=554454]: Done uploading segments... Jan 1 00:13:52.601627638 (632h44m52.472710673s) [upload=554454]: Upload finished. err: <nil> Duration: 5.975375034s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X............X....X....X......X....X....X.....XXX..X..XX...] duration: 16.229269012s trace: https://jaeger.dev.storj.io/trace/28a9125ed5f2f5b5 FAIL (2026-01-01 00:16:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Jan 1 00:13:49.620961485 (632h44m49.49204452s) [upload=554454]: Got next segment. Inline: false Jan 1 00:13:49.620997573 (632h44m49.492080618s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:13:49.621002192 (632h44m49.492085237s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:13:49.868640814 (632h44m49.739723849s) [seg_pos={0 0}, upload=554454]: Begin upload segment... Jan 1 00:13:49.868705798 (632h44m49.739788833s) [seg_pos={0 0}, upload=554454]: Done begin upload segment. Jan 1 00:13:49.868724784 (632h44m49.739807819s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868748168 (632h44m49.739831203s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868826719 (632h44m49.739909764s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869134614 (632h44m49.740217659s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869399539 (632h44m49.740482574s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870161510 (632h44m49.741244545s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870452644 (632h44m49.741535679s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870806107 (632h44m49.741889152s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871111188 (632h44m49.742194223s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871466394 (632h44m49.742549429s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872254716 (632h44m49.743337751s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872531743 (632h44m49.743614788s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.873624453 (632h44m49.744707489s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.874767320 (632h44m49.745850355s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.875797311 (632h44m49.746880346s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.876955366 (632h44m49.748038401s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:51.575471743 (632h44m51.446554788s) [upload=554454]: Next returned nil segment Jan 1 00:13:51.575676152 (632h44m51.446759227s) [upload=554454]: Waiting for error group managing segments... Jan 1 00:13:51.855996814 (632h44m51.727079849s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.864465159 (632h44m51.735548204s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.868352721 (632h44m51.739435756s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.881984337 (632h44m51.753067382s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.884508242 (632h44m51.755591277s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.896797463 (632h44m51.767880498s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.942212709 (632h44m51.813295744s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.947553388 (632h44m51.818636443s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.017191837 (632h44m51.888274882s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.157166414 (632h44m52.028249469s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.166689146 (632h44m52.037772192s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.240675014 (632h44m52.111758059s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301683050 (632h44m52.172766095s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301707176 (632h44m52.172790222s) [seg_pos={0 0}, upload=554454]: Segment reached optimal threshold of 13 pieces. Jan 1 00:13:52.301980036 (632h44m52.173063081s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:52.301991337 (632h44m52.173074372s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302201427 (632h44m52.173284472s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection Jan 1 00:13:52.302211015 (632h44m52.17329406s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302356243 (632h44m52.173439278s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection Jan 1 00:13:52.302366773 (632h44m52.173449808s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302390067 (632h44m52.173473102s) [seg_pos={0 0}, upload=554454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:13:52.302479958 (632h44m52.173563003s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:52.302487141 (632h44m52.173570186s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:52.601605125 (632h44m52.47268817s) [upload=554454]: Done uploading segments... Jan 1 00:13:52.601627638 (632h44m52.472710673s) [upload=554454]: Upload finished. err: <nil> Duration: 16.229302266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X....................X...........X..X..] duration: 39.491118014s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:16:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 39.491159463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X............X....X....X......X....X....X.....XXX..X..XX..] duration: 5.973357132s trace: https://jaeger.dev.storj.io/trace/28a9125ed5f2f5b5 FAIL (2026-01-01 00:15:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Jan 1 00:13:49.620961485 (632h44m49.49204452s) [upload=554454]: Got next segment. Inline: false Jan 1 00:13:49.620997573 (632h44m49.492080618s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:13:49.621002192 (632h44m49.492085237s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:13:49.868640814 (632h44m49.739723849s) [seg_pos={0 0}, upload=554454]: Begin upload segment... Jan 1 00:13:49.868705798 (632h44m49.739788833s) [seg_pos={0 0}, upload=554454]: Done begin upload segment. Jan 1 00:13:49.868724784 (632h44m49.739807819s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868748168 (632h44m49.739831203s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868826719 (632h44m49.739909764s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869134614 (632h44m49.740217659s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869399539 (632h44m49.740482574s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870161510 (632h44m49.741244545s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870452644 (632h44m49.741535679s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870806107 (632h44m49.741889152s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871111188 (632h44m49.742194223s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871466394 (632h44m49.742549429s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872254716 (632h44m49.743337751s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872531743 (632h44m49.743614788s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.873624453 (632h44m49.744707489s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.874767320 (632h44m49.745850355s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.875797311 (632h44m49.746880346s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.876955366 (632h44m49.748038401s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:51.575471743 (632h44m51.446554788s) [upload=554454]: Next returned nil segment Jan 1 00:13:51.575676152 (632h44m51.446759227s) [upload=554454]: Waiting for error group managing segments... Jan 1 00:13:51.855996814 (632h44m51.727079849s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.864465159 (632h44m51.735548204s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.868352721 (632h44m51.739435756s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.881984337 (632h44m51.753067382s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.884508242 (632h44m51.755591277s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.896797463 (632h44m51.767880498s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.942212709 (632h44m51.813295744s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.947553388 (632h44m51.818636443s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.017191837 (632h44m51.888274882s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.157166414 (632h44m52.028249469s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.166689146 (632h44m52.037772192s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.240675014 (632h44m52.111758059s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301683050 (632h44m52.172766095s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301707176 (632h44m52.172790222s) [seg_pos={0 0}, upload=554454]: Segment reached optimal threshold of 13 pieces. Jan 1 00:13:52.301980036 (632h44m52.173063081s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:52.301991337 (632h44m52.173074372s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302201427 (632h44m52.173284472s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection Jan 1 00:13:52.302211015 (632h44m52.17329406s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302356243 (632h44m52.173439278s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection Jan 1 00:13:52.302366773 (632h44m52.173449808s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302390067 (632h44m52.173473102s) [seg_pos={0 0}, upload=554454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:13:52.302479958 (632h44m52.173563003s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:52.302487141 (632h44m52.173570186s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:52.601605125 (632h44m52.47268817s) [upload=554454]: Done uploading segments... Jan 1 00:13:52.601627638 (632h44m52.472710673s) [upload=554454]: Upload finished. err: <nil> Duration: 5.973382963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X............X....X....X......X....X....X.....XXX..X..XX.] duration: 15.29617228s trace: https://jaeger.dev.storj.io/trace/28a9125ed5f2f5b5 FAIL (2026-01-01 00:15:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Jan 1 00:13:49.620961485 (632h44m49.49204452s) [upload=554454]: Got next segment. Inline: false Jan 1 00:13:49.620997573 (632h44m49.492080618s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:13:49.621002192 (632h44m49.492085237s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:13:49.868640814 (632h44m49.739723849s) [seg_pos={0 0}, upload=554454]: Begin upload segment... Jan 1 00:13:49.868705798 (632h44m49.739788833s) [seg_pos={0 0}, upload=554454]: Done begin upload segment. Jan 1 00:13:49.868724784 (632h44m49.739807819s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868748168 (632h44m49.739831203s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868826719 (632h44m49.739909764s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869134614 (632h44m49.740217659s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869399539 (632h44m49.740482574s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870161510 (632h44m49.741244545s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870452644 (632h44m49.741535679s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870806107 (632h44m49.741889152s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871111188 (632h44m49.742194223s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871466394 (632h44m49.742549429s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872254716 (632h44m49.743337751s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872531743 (632h44m49.743614788s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.873624453 (632h44m49.744707489s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.874767320 (632h44m49.745850355s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.875797311 (632h44m49.746880346s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.876955366 (632h44m49.748038401s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:51.575471743 (632h44m51.446554788s) [upload=554454]: Next returned nil segment Jan 1 00:13:51.575676152 (632h44m51.446759227s) [upload=554454]: Waiting for error group managing segments... Jan 1 00:13:51.855996814 (632h44m51.727079849s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.864465159 (632h44m51.735548204s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.868352721 (632h44m51.739435756s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.881984337 (632h44m51.753067382s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.884508242 (632h44m51.755591277s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.896797463 (632h44m51.767880498s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.942212709 (632h44m51.813295744s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.947553388 (632h44m51.818636443s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.017191837 (632h44m51.888274882s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.157166414 (632h44m52.028249469s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.166689146 (632h44m52.037772192s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.240675014 (632h44m52.111758059s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301683050 (632h44m52.172766095s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301707176 (632h44m52.172790222s) [seg_pos={0 0}, upload=554454]: Segment reached optimal threshold of 13 pieces. Jan 1 00:13:52.301980036 (632h44m52.173063081s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:52.301991337 (632h44m52.173074372s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302201427 (632h44m52.173284472s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection Jan 1 00:13:52.302211015 (632h44m52.17329406s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302356243 (632h44m52.173439278s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection Jan 1 00:13:52.302366773 (632h44m52.173449808s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302390067 (632h44m52.173473102s) [seg_pos={0 0}, upload=554454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:13:52.302479958 (632h44m52.173563003s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:52.302487141 (632h44m52.173570186s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:52.601605125 (632h44m52.47268817s) [upload=554454]: Done uploading segments... Jan 1 00:13:52.601627638 (632h44m52.472710673s) [upload=554454]: Upload finished. err: <nil> Duration: 15.296203289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X....................X...........X..X.] duration: 30.080124989s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:14:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 30.080139185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X....................X...........X..X] duration: 1m0.004248211s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:14:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 1m0.004265413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...X............X....X....X......X....X....X.....XXX..X..XX] duration: 30.002523798s trace: https://jaeger.dev.storj.io/trace/28a9125ed5f2f5b5 FAIL (2026-01-01 00:14:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Jan 1 00:13:49.620961485 (632h44m49.49204452s) [upload=554454]: Got next segment. Inline: false Jan 1 00:13:49.620997573 (632h44m49.492080618s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:13:49.621002192 (632h44m49.492085237s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:13:49.868640814 (632h44m49.739723849s) [seg_pos={0 0}, upload=554454]: Begin upload segment... Jan 1 00:13:49.868705798 (632h44m49.739788833s) [seg_pos={0 0}, upload=554454]: Done begin upload segment. Jan 1 00:13:49.868724784 (632h44m49.739807819s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868748168 (632h44m49.739831203s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.868826719 (632h44m49.739909764s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869134614 (632h44m49.740217659s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.869399539 (632h44m49.740482574s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870161510 (632h44m49.741244545s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870452644 (632h44m49.741535679s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.870806107 (632h44m49.741889152s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871111188 (632h44m49.742194223s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.871466394 (632h44m49.742549429s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872254716 (632h44m49.743337751s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.872531743 (632h44m49.743614788s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.873624453 (632h44m49.744707489s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.874767320 (632h44m49.745850355s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.875797311 (632h44m49.746880346s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:49.876955366 (632h44m49.748038401s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Uploading piece... Jan 1 00:13:51.575471743 (632h44m51.446554788s) [upload=554454]: Next returned nil segment Jan 1 00:13:51.575676152 (632h44m51.446759227s) [upload=554454]: Waiting for error group managing segments... Jan 1 00:13:51.855996814 (632h44m51.727079849s) [noise=true , address=145.79.227.6:4259 , piece_id=6DAY6QZXYJTZE2RTP7JVGUJPVINM3AIIZ657JK63TYG5HFGIWU5Q, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.864465159 (632h44m51.735548204s) [noise=true , address=145.79.227.36:4519 , piece_id=WMR3QCG2ESCGQONU2U37S2AM6SC2CKCZZIEAFRY6N5AYPQIAQV7A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.868352721 (632h44m51.739435756s) [noise=true , address=145.79.227.27:4689 , piece_id=ZQB5KQHPDNJSGEQ6ISOZ7FAAFBZIKFR5R2QUZS64TWBJWCIUVVRQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.881984337 (632h44m51.753067382s) [noise=true , address=145.79.227.52:4445 , piece_id=UCWMSGKZNVX7YQBFS5THXQ4CY3WHN3YZDD6GVT57GMZ4R6IHCPIA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.884508242 (632h44m51.755591277s) [noise=true , address=145.79.227.18:4006 , piece_id=OBQFO5U4EAN77BXSDMTUBKEAWRZCCH6JKRSHJ4TVUFCPCJ5VN3DA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.896797463 (632h44m51.767880498s) [noise=true , address=145.79.227.13:4804 , piece_id=Q7LKDBEXOCAMZA3FOESRKQDEFTI22YATKSFRBI3EAIFEBZ2A23CQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.942212709 (632h44m51.813295744s) [noise=true , address=145.79.227.17:4972 , piece_id=PCY4VHPXJLGDDXPYBUHBQB5SG3L53Y7I266IONPT74SXI6Z6X5YA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:51.947553388 (632h44m51.818636443s) [noise=true , address=145.79.227.11:4732 , piece_id=BAE62OAYWS43GTQ4SMWMVYSEI7YB2LDZQCHM2ZPZ6BM6FDI4V56A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.017191837 (632h44m51.888274882s) [noise=true , address=77.90.31.13:4627 , piece_id=SLAPX6O6PITQVTX4TJJT7YEHSLYJXHHXPLHBZ3VECDR4NVQSBLZQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.157166414 (632h44m52.028249469s) [noise=true , address=45.91.116.14:4189 , piece_id=GFWWQQGAKVXT3UEDLVICBTSW7TH7JLSW6XCVAIGHFGQOC3ZVWYDQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.166689146 (632h44m52.037772192s) [noise=true , address=77.90.31.11:4553 , piece_id=RZFG2JKGQY4FBHWLI7BNPWPN5RTLMASDUFXO3U5NB5Z3AZUOVN2A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.240675014 (632h44m52.111758059s) [noise=true , address=45.91.116.7:4011 , piece_id=TLS4QVOCFB7OBOZ5JQYR77VNZGSUML6DUW7SS3X42W3ANXZV5RHQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301683050 (632h44m52.172766095s) [noise=true , address=45.91.116.8:4047 , piece_id=XZJZKA5Z6K4LFRGJCB5LJPN35X2XBW5D3OEXV6A23DL7ZOLOOXVA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: <nil> Jan 1 00:13:52.301707176 (632h44m52.172790222s) [seg_pos={0 0}, upload=554454]: Segment reached optimal threshold of 13 pieces. Jan 1 00:13:52.301980036 (632h44m52.173063081s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:52.301991337 (632h44m52.173074372s) [noise=true , address=145.79.227.49:4337 , piece_id=R7WIK3YTKBGVIZ4BQW7QUK54UAX52OMTICPKSDYMUFCQB4WM6A6A, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302201427 (632h44m52.173284472s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43496->77.90.31.10:4517: use of closed network connection Jan 1 00:13:52.302211015 (632h44m52.17329406s) [noise=true , address=77.90.31.10:4517 , piece_id=KBSNXRQDPSLIQQCXX5A63JXAZFD25ULI4KTK5ZKVDUBS46L6HZCA, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302356243 (632h44m52.173439278s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57692->45.91.116.5:4298: use of closed network connection Jan 1 00:13:52.302366773 (632h44m52.173449808s) [noise=true , address=45.91.116.5:4298 , piece_id=IOR3OA4C5W7KPGYRCYCGTRTHAVILOVRRVXUU7HT2J72Q4K6TIFXQ, seg_pos={0 0}, upload=554454]: Long tail cancellation due to optimal threshold Jan 1 00:13:52.302390067 (632h44m52.173473102s) [seg_pos={0 0}, upload=554454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:13:52.302479958 (632h44m52.173563003s) [upload=554454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:52.302487141 (632h44m52.173570186s) [upload=554454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:52.601605125 (632h44m52.47268817s) [upload=554454]: Done uploading segments... Jan 1 00:13:52.601627638 (632h44m52.472710673s) [upload=554454]: Upload finished. err: <nil> Duration: 30.002534809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X...X............X....X....X......X....X....X.....XXX..X..X] duration: 30.00174422s trace: https://jaeger.dev.storj.io/trace/214e771d9ad8d076 FAIL (2026-01-01 00:13:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: L7UVU6QFSTS5LNPYUA6NCXBOFAIQMM6ZRTNKV6JYN4Y62YG5EPWQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: JBTKTHQNUR7TYR4HMB3Y7HM6LNVJBGKI6MJTO7QVXKBGDPE5YKQA) context deadline exceeded ======LOGS====== Jan 1 00:13:19.368193035 (632h44m19.23927608s) [upload=554444]: Starting upload Jan 1 00:13:19.368253169 (632h44m19.239336204s) [upload=554444]: Done starting upload Jan 1 00:13:19.368385441 (632h44m19.239468486s) [upload=554444]: Uploading segments... Jan 1 00:13:19.368474961 (632h44m19.239557997s) [upload=554444]: Got next segment. Inline: false Jan 1 00:13:19.368531609 (632h44m19.239614644s) [upload=554444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:13:19.368538002 (632h44m19.239621047s) [upload=554444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:13:19.616712293 (632h44m19.487795328s) [seg_pos={0 0}, upload=554444]: Begin upload segment... Jan 1 00:13:19.616759844 (632h44m19.487842879s) [seg_pos={0 0}, upload=554444]: Done begin upload segment. Jan 1 00:13:19.616790281 (632h44m19.487873316s) [noise=true , address=77.90.31.17:4773 , piece_id=JBTKTHQNUR7TYR4HMB3Y7HM6LNVJBGKI6MJTO7QVXKBGDPE5YKQA, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.616964503 (632h44m19.488047538s) [noise=true , address=145.79.227.31:4798 , piece_id=4BDPOG224RQMXIV4G66GZFMTL2BIJUESAIMNXIB2UZ33LA6GLWHQ, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.617759728 (632h44m19.488842763s) [noise=true , address=145.79.227.8:4329 , piece_id=LTIDY4PPKZCBFTFCTJPKB76462ZESJ3P6BQHZXCD32B4HBC4RG3A, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.617869316 (632h44m19.488952351s) [noise=true , address=145.79.227.4:4185 , piece_id=VXZM3KA4TGZ4UXPKI436IFKDTDFHQR7FSXTHCBCBVHTT6DXUPKDA, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.618176060 (632h44m19.489259095s) [noise=true , address=145.79.227.13:4802 , piece_id=B5HINAD57AM3SB4QJME7PAHOOJYJ53P5UKS2YVYUDFCFPD3POFBA, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.618332509 (632h44m19.489415554s) [noise=true , address=145.79.227.9:4366 , piece_id=42BE6YGPHUBHYIXEAWYY6NUPHFWDC4ENTCEATQYP5A2GJQJGCZWA, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.618436627 (632h44m19.489519672s) [noise=true , address=45.91.116.8:4046 , piece_id=CW7VE4RUWXYIDS5XIOVVGVMMVAXLZX773DVUT2R7D3SQ3A6MFV5Q, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.618772646 (632h44m19.489855681s) [noise=true , address=145.79.227.21:4474 , piece_id=OAX325OC6GRYDNFFDBWO6QJ3LQNDMZPLUKCZC5RYQAUMXORKWNZA, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.618968389 (632h44m19.490051434s) [noise=true , address=145.79.227.15:4876 , piece_id=HA2SSOO4OJQZCCA5T6CR5CT6B4EVWSPWR4VFYNKHSRII2TNHMWLA, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.619051678 (632h44m19.490134713s) [noise=true , address=145.79.227.26:4655 , piece_id=DHZLH62IHN4G57SRGRMA5OAQLYHQG5D2WUNHHVL2SQN4FVQRQLKA, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.619333093 (632h44m19.490416128s) [noise=true , address=77.90.31.21:4878 , piece_id=L7UVU6QFSTS5LNPYUA6NCXBOFAIQMM6ZRTNKV6JYN4Y62YG5EPWQ, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.619522024 (632h44m19.490605059s) [noise=true , address=145.79.227.52:4445 , piece_id=LWAMD56S3JVJEEYHCZXC55V266TQQSHFTWT36HBRL4IZFV5TWE6A, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.619602146 (632h44m19.490685181s) [noise=true , address=45.91.116.6:4976 , piece_id=QSPOGLIF27MSLEMGSF2KUVB3GWIP3D36UYVSLKLFYA5B5A3JMXAA, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.620106135 (632h44m19.49118918s) [noise=true , address=45.91.116.11:4118 , piece_id=76W7RAKERUUU7OUS4OZBRAXWD4GU2E3AY74XGNCRVIJITPURSV3Q, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.620719734 (632h44m19.491802769s) [noise=true , address=77.90.31.14:4664 , piece_id=RCJXNJXMAVRWEGX2AQXJWGRPVR2WS5ARNU3MAW25A2BCWOP6KIYQ, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:19.621576795 (632h44m19.49265984s) [noise=true , address=145.79.227.7:4295 , piece_id=ZASYAR7O5Q2WJ3HLUDKY6GPZJR6GFRJMI7LCR54WCGHVH5XWNJ6A, seg_pos={0 0}, upload=554444]: Uploading piece... Jan 1 00:13:21.326784834 (632h44m21.197867869s) [upload=554444]: Next returned nil segment Jan 1 00:13:21.326819209 (632h44m21.197902254s) [upload=554444]: Waiting for error group managing segments... Jan 1 00:13:21.607859111 (632h44m21.478942147s) [noise=true , address=145.79.227.52:4445 , piece_id=LWAMD56S3JVJEEYHCZXC55V266TQQSHFTWT36HBRL4IZFV5TWE6A, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.609264277 (632h44m21.480347312s) [noise=true , address=145.79.227.15:4876 , piece_id=HA2SSOO4OJQZCCA5T6CR5CT6B4EVWSPWR4VFYNKHSRII2TNHMWLA, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.611552224 (632h44m21.482635259s) [noise=true , address=145.79.227.4:4185 , piece_id=VXZM3KA4TGZ4UXPKI436IFKDTDFHQR7FSXTHCBCBVHTT6DXUPKDA, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.625471838 (632h44m21.496554873s) [noise=true , address=145.79.227.21:4474 , piece_id=OAX325OC6GRYDNFFDBWO6QJ3LQNDMZPLUKCZC5RYQAUMXORKWNZA, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.628716957 (632h44m21.499800002s) [noise=true , address=145.79.227.26:4655 , piece_id=DHZLH62IHN4G57SRGRMA5OAQLYHQG5D2WUNHHVL2SQN4FVQRQLKA, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.629342378 (632h44m21.500425423s) [noise=true , address=145.79.227.31:4798 , piece_id=4BDPOG224RQMXIV4G66GZFMTL2BIJUESAIMNXIB2UZ33LA6GLWHQ, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.629739123 (632h44m21.500822158s) [noise=true , address=145.79.227.9:4366 , piece_id=42BE6YGPHUBHYIXEAWYY6NUPHFWDC4ENTCEATQYP5A2GJQJGCZWA, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.630340228 (632h44m21.501423263s) [noise=true , address=145.79.227.7:4295 , piece_id=ZASYAR7O5Q2WJ3HLUDKY6GPZJR6GFRJMI7LCR54WCGHVH5XWNJ6A, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.630980507 (632h44m21.502063542s) [noise=true , address=145.79.227.8:4329 , piece_id=LTIDY4PPKZCBFTFCTJPKB76462ZESJ3P6BQHZXCD32B4HBC4RG3A, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.790992949 (632h44m21.662075984s) [noise=true , address=77.90.31.21:4878 , piece_id=L7UVU6QFSTS5LNPYUA6NCXBOFAIQMM6ZRTNKV6JYN4Y62YG5EPWQ, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.808797661 (632h44m21.679880706s) [noise=true , address=77.90.31.14:4664 , piece_id=RCJXNJXMAVRWEGX2AQXJWGRPVR2WS5ARNU3MAW25A2BCWOP6KIYQ, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.827339277 (632h44m21.698422322s) [noise=true , address=145.79.227.13:4802 , piece_id=B5HINAD57AM3SB4QJME7PAHOOJYJ53P5UKS2YVYUDFCFPD3POFBA, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.842058844 (632h44m21.713141879s) [noise=true , address=77.90.31.17:4773 , piece_id=JBTKTHQNUR7TYR4HMB3Y7HM6LNVJBGKI6MJTO7QVXKBGDPE5YKQA, seg_pos={0 0}, upload=554444]: Done uploading piece. err: <nil> Jan 1 00:13:21.842076548 (632h44m21.713159593s) [seg_pos={0 0}, upload=554444]: Segment reached optimal threshold of 13 pieces. Jan 1 00:13:21.842309752 (632h44m21.713392787s) [noise=true , address=45.91.116.11:4118 , piece_id=76W7RAKERUUU7OUS4OZBRAXWD4GU2E3AY74XGNCRVIJITPURSV3Q, seg_pos={0 0}, upload=554444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:21.842318007 (632h44m21.713401043s) [noise=true , address=45.91.116.11:4118 , piece_id=76W7RAKERUUU7OUS4OZBRAXWD4GU2E3AY74XGNCRVIJITPURSV3Q, seg_pos={0 0}, upload=554444]: Long tail cancellation due to optimal threshold Jan 1 00:13:21.842462302 (632h44m21.713545337s) [noise=true , address=45.91.116.8:4046 , piece_id=CW7VE4RUWXYIDS5XIOVVGVMMVAXLZX773DVUT2R7D3SQ3A6MFV5Q, seg_pos={0 0}, upload=554444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:21.842477451 (632h44m21.713560487s) [noise=true , address=45.91.116.8:4046 , piece_id=CW7VE4RUWXYIDS5XIOVVGVMMVAXLZX773DVUT2R7D3SQ3A6MFV5Q, seg_pos={0 0}, upload=554444]: Long tail cancellation due to optimal threshold Jan 1 00:13:21.842618851 (632h44m21.713701886s) [noise=true , address=45.91.116.6:4976 , piece_id=QSPOGLIF27MSLEMGSF2KUVB3GWIP3D36UYVSLKLFYA5B5A3JMXAA, seg_pos={0 0}, upload=554444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:21.842626455 (632h44m21.71370949s) [noise=true , address=45.91.116.6:4976 , piece_id=QSPOGLIF27MSLEMGSF2KUVB3GWIP3D36UYVSLKLFYA5B5A3JMXAA, seg_pos={0 0}, upload=554444]: Long tail cancellation due to optimal threshold Jan 1 00:13:21.842643787 (632h44m21.713726823s) [seg_pos={0 0}, upload=554444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:13:21.842738959 (632h44m21.713821994s) [upload=554444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:21.842746563 (632h44m21.713829608s) [upload=554444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:22.131855670 (632h44m22.002938725s) [upload=554444]: Done uploading segments... Jan 1 00:13:22.131878794 (632h44m22.002961839s) [upload=554444]: Upload finished. err: <nil> Duration: 30.001763568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X...X............X....X....X......X....X....X.....XXX..X..] duration: 6.048258079s trace: https://jaeger.dev.storj.io/trace/53dd760923edc3ff FAIL (2026-01-01 00:12:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: UWMHSXVKZ6STQZ5LC4RDNOWXRAYTNLA76JZ7D7GUTQCTTN7QRC3Q) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: IE7ERGANAPI2VUVHZXLUULV5LU6E7MUPLSS5BOWUXDONDVIZ5XDQ) context deadline exceeded ======LOGS====== 1 00:11:41.949952472 (632h42m41.821035517s) [upload=554420]: Starting upload Jan 1 00:11:41.950091206 (632h42m41.821174241s) [upload=554420]: Done starting upload Jan 1 00:11:41.950192439 (632h42m41.821275484s) [upload=554420]: Uploading segments... Jan 1 00:11:41.950505234 (632h42m41.821588279s) [upload=554420]: Got next segment. Inline: false Jan 1 00:11:41.950605825 (632h42m41.82168887s) [upload=554420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:11:41.950613130 (632h42m41.821696175s) [upload=554420]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:11:42.195838456 (632h42m42.066921491s) [seg_pos={0 0}, upload=554420]: Begin upload segment... Jan 1 00:11:42.195885456 (632h42m42.066968491s) [seg_pos={0 0}, upload=554420]: Done begin upload segment. Jan 1 00:11:42.195910192 (632h42m42.066993237s) [noise=true , address=45.91.116.6:4976 , piece_id=AKBTGAYVC2NJRTIJX3VCHQTE7SXKTU5SJP7GBK3C6MCJ3PGYXHPQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.196116836 (632h42m42.067199871s) [noise=true , address=77.90.31.14:4663 , piece_id=UWMHSXVKZ6STQZ5LC4RDNOWXRAYTNLA76JZ7D7GUTQCTTN7QRC3Q, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.196537357 (632h42m42.067620392s) [noise=true , address=45.91.116.10:4082 , piece_id=E6ACTYUL4VAYYXCAHG7GMUFAGUXW7NOAJNTXTN2GSQCTTTFCHQEA, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.197151987 (632h42m42.068235032s) [noise=true , address=77.90.31.10:4518 , piece_id=FZR2XRCKGUKH4M3QXY7YG3FFP5LTL4JIM6H4JHGHPOPI4YQLBJYA, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.197636189 (632h42m42.068719224s) [noise=true , address=145.79.227.16:4912 , piece_id=ZG3ZPCTOFITPO53CBT2XYVQ2R2WPKKOQ6DWPNRY6QRGD3RJILYXQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.197654684 (632h42m42.068737729s) [noise=true , address=45.91.116.7:4012 , piece_id=WTMUGF36HNL2XZLD74QFQQ7RYQEKANMOWOVOKDBXF73CABWBD6IQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.198176117 (632h42m42.069259162s) [noise=true , address=145.79.227.38:4978 , piece_id=RGFS23W3GXKIPAM2QEUKPHPA73ZIMAQQPBZAPXAL4GN4REL2ESMQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.198785267 (632h42m42.069868313s) [noise=true , address=45.91.116.13:4154 , piece_id=IE7ERGANAPI2VUVHZXLUULV5LU6E7MUPLSS5BOWUXDONDVIZ5XDQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.199689439 (632h42m42.070772474s) [noise=true , address=145.79.227.19:4077 , piece_id=JYQDFVN3TGJMGARIE4CXWCMNMOZO2BNLB3XGAORIARA4R4N3KGHQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.201012999 (632h42m42.072096034s) [noise=true , address=145.79.227.9:4365 , piece_id=7URIT33NM7EUARFJZCRJDLQHB7SI7PQ7OYL6DQXFJIROJRXZOOWQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.201412670 (632h42m42.072495705s) [noise=true , address=145.79.227.11:4730 , piece_id=ZM23S4LC4C6H3PZC4WHBK7R57MCZZSFNAI6R6IM3UTGUVRTRD6IQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.201817722 (632h42m42.072900757s) [noise=true , address=145.79.227.3:4151 , piece_id=FCT37VLJQ3K334JGRHAZTL6IEFLBXCJ6E4AD6ANIA2YJVCSPIMSQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.202401594 (632h42m42.073484629s) [noise=true , address=145.79.227.15:4873 , piece_id=6XSMNC5455BL7LXNEMWDZAWD4LNTYNYU3VGL56ZBGRM77LIJ6DDQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.202856600 (632h42m42.073939645s) [noise=true , address=145.79.227.50:4373 , piece_id=6BUP2CXGVY3KSDMG2ZZF4QNNAIHSSGBFL72T4L2OUNH4EHRDLOYA, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.204577476 (632h42m42.075660521s) [noise=true , address=145.79.227.24:4583 , piece_id=JRMHJEBQEDULMLRANAZUQQ544UG5OHXNJGAO56R6TGF75SVCUNYQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.207128464 (632h42m42.078211499s) [noise=true , address=145.79.227.10:4401 , piece_id=M53H7CHT2NDVXPE7HRZIUNF64BH46YSVKDLK2EKE2W6UQR7M2KNQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:43.894621269 (632h42m43.765704304s) [upload=554420]: Next returned nil segment Jan 1 00:11:43.894641658 (632h42m43.765724704s) [upload=554420]: Waiting for error group managing segments... Jan 1 00:11:44.177169193 (632h42m44.048252238s) [noise=true , address=145.79.227.11:4730 , piece_id=ZM23S4LC4C6H3PZC4WHBK7R57MCZZSFNAI6R6IM3UTGUVRTRD6IQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.177604331 (632h42m44.048687366s) [noise=true , address=145.79.227.50:4373 , piece_id=6BUP2CXGVY3KSDMG2ZZF4QNNAIHSSGBFL72T4L2OUNH4EHRDLOYA, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.191346609 (632h42m44.062429644s) [noise=true , address=145.79.227.19:4077 , piece_id=JYQDFVN3TGJMGARIE4CXWCMNMOZO2BNLB3XGAORIARA4R4N3KGHQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.193352708 (632h42m44.064435753s) [noise=true , address=145.79.227.3:4151 , piece_id=FCT37VLJQ3K334JGRHAZTL6IEFLBXCJ6E4AD6ANIA2YJVCSPIMSQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.193746588 (632h42m44.064829623s) [noise=true , address=145.79.227.15:4873 , piece_id=6XSMNC5455BL7LXNEMWDZAWD4LNTYNYU3VGL56ZBGRM77LIJ6DDQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.198818185 (632h42m44.06990123s) [noise=true , address=145.79.227.9:4365 , piece_id=7URIT33NM7EUARFJZCRJDLQHB7SI7PQ7OYL6DQXFJIROJRXZOOWQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.211799272 (632h42m44.082882318s) [noise=true , address=145.79.227.38:4978 , piece_id=RGFS23W3GXKIPAM2QEUKPHPA73ZIMAQQPBZAPXAL4GN4REL2ESMQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.217579309 (632h42m44.088662354s) [noise=true , address=145.79.227.16:4912 , piece_id=ZG3ZPCTOFITPO53CBT2XYVQ2R2WPKKOQ6DWPNRY6QRGD3RJILYXQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.218513196 (632h42m44.089596261s) [noise=true , address=145.79.227.24:4583 , piece_id=JRMHJEBQEDULMLRANAZUQQ544UG5OHXNJGAO56R6TGF75SVCUNYQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.220628164 (632h42m44.091711209s) [noise=true , address=145.79.227.10:4401 , piece_id=M53H7CHT2NDVXPE7HRZIUNF64BH46YSVKDLK2EKE2W6UQR7M2KNQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.440555750 (632h42m44.311638785s) [noise=true , address=77.90.31.14:4663 , piece_id=UWMHSXVKZ6STQZ5LC4RDNOWXRAYTNLA76JZ7D7GUTQCTTN7QRC3Q, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.501467963 (632h42m44.372550998s) [noise=true , address=45.91.116.13:4154 , piece_id=IE7ERGANAPI2VUVHZXLUULV5LU6E7MUPLSS5BOWUXDONDVIZ5XDQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.588674244 (632h42m44.459757289s) [noise=true , address=45.91.116.7:4012 , piece_id=WTMUGF36HNL2XZLD74QFQQ7RYQEKANMOWOVOKDBXF73CABWBD6IQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.588696677 (632h42m44.459779722s) [seg_pos={0 0}, upload=554420]: Segment reached optimal threshold of 13 pieces. Jan 1 00:11:44.588904032 (632h42m44.459987067s) [noise=true , address=45.91.116.6:4976 , piece_id=AKBTGAYVC2NJRTIJX3VCHQTE7SXKTU5SJP7GBK3C6MCJ3PGYXHPQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 00:11:44.588911256 (632h42m44.459994291s) [noise=true , address=45.91.116.6:4976 , piece_id=AKBTGAYVC2NJRTIJX3VCHQTE7SXKTU5SJP7GBK3C6MCJ3PGYXHPQ, seg_pos={0 0}, upload=554420]: Long tail cancellation due to optimal threshold Jan 1 00:11:44.588962394 (632h42m44.460045439s) [noise=true , address=77.90.31.10:4518 , piece_id=FZR2XRCKGUKH4M3QXY7YG3FFP5LTL4JIM6H4JHGHPOPI4YQLBJYA, seg_pos={0 0}, upload=554420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:34074->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:34074->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34074->77.90.31.10:4518: use of closed network connection Jan 1 00:11:44.588972573 (632h42m44.460055618s) [noise=true , address=77.90.31.10:4518 , piece_id=FZR2XRCKGUKH4M3QXY7YG3FFP5LTL4JIM6H4JHGHPOPI4YQLBJYA, seg_pos={0 0}, upload=554420]: Long tail cancellation due to optimal threshold Jan 1 00:11:44.589162293 (632h42m44.460245328s) [noise=true , address=45.91.116.10:4082 , piece_id=E6ACTYUL4VAYYXCAHG7GMUFAGUXW7NOAJNTXTN2GSQCTTTFCHQEA, seg_pos={0 0}, upload=554420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 1 00:11:44.589169838 (632h42m44.460252883s) [noise=true , address=45.91.116.10:4082 , piece_id=E6ACTYUL4VAYYXCAHG7GMUFAGUXW7NOAJNTXTN2GSQCTTTFCHQEA, seg_pos={0 0}, upload=554420]: Long tail cancellation due to optimal threshold Jan 1 00:11:44.589187091 (632h42m44.460270136s) [seg_pos={0 0}, upload=554420]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:11:44.589266151 (632h42m44.460349206s) [upload=554420]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:11:44.589271551 (632h42m44.460354586s) [upload=554420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:11:44.892025329 (632h42m44.763108364s) [upload=554420]: Done uploading segments... Jan 1 00:11:44.892050196 (632h42m44.763133241s) [upload=554420]: Upload finished. err: <nil> Duration: 6.048291102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X....................X...........X..] duration: 33.81055149s trace: https://jaeger.dev.storj.io/trace/67e2279657e45584 FAIL (2026-01-01 00:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, Piece ID: Y4TNTQZ2DERVYJ45IWES34IREZM72VWEUJT5DFXBPSFXCXT2ZLBA) context deadline exceeded; piecestore: (Node ID: 12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, Piece ID: 2SZZCPGVL77TPTSZCD7YPVJSFVKJGQHYQAMMGOP23VMEELC6DNSA) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: EJOPAXWICBWSBGS4USQCMHX6U4XWECUEPRJJ3RQIPAMHULYOXOFQ) context deadline exceeded; piecestore: (Node ID: 12ZNCRKd7xMFNHfCJBJ38yfyUmczhYwJetKtiQxX9SDrwxRhzTU, Piece ID: S7XHBNCFCHXSC6XEBXTKXGQV2C3QYOZUNJHXUK32MV62OR7THHYA) context deadline exceeded ======LOGS====== :4936 , piece_id=REO4SYQE655A6OWXGR2IARSVCLQ3OS24DDIQJS5ZSL5VO33ZMGFQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.050418220 (632h41m18.921501255s) [noise=true , address=77.90.31.14:4677 , piece_id=EJOPAXWICBWSBGS4USQCMHX6U4XWECUEPRJJ3RQIPAMHULYOXOFQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.050930676 (632h41m18.922013721s) [noise=true , address=77.90.31.15:4714 , piece_id=Y4TNTQZ2DERVYJ45IWES34IREZM72VWEUJT5DFXBPSFXCXT2ZLBA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.051454794 (632h41m18.922537839s) [noise=true , address=45.91.116.12:4251 , piece_id=S7XHBNCFCHXSC6XEBXTKXGQV2C3QYOZUNJHXUK32MV62OR7THHYA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.051881115 (632h41m18.92296416s) [noise=true , address=64.49.13.140:4509 , piece_id=DR2NS5DIRRU4LHOL7VEIAQZLDM6IPBPCCFULLZSFRJOPVUH36DJQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.052297879 (632h41m18.923380924s) [noise=true , address=64.49.13.140:4511 , piece_id=3HN3335O7DAFTBJA5KQP6UXWZVPZ5VUIB3EFA4NJJWBXRXFJF7JQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.052727466 (632h41m18.923810502s) [noise=true , address=77.90.31.11:4581 , piece_id=NNTFVIMH65B65LCG4B33Y5PW3M2ZTBWM5EPJN7LCXFBIW65C4P3Q, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.053205347 (632h41m18.924288392s) [noise=true , address=145.79.227.28:4760 , piece_id=VELHSBW4LSXCQVAXKZ3XOKF2WNVCL7VUEKB7N6UKKLA5ZPXCVHGA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.054372899 (632h41m18.925455954s) [noise=true , address=77.90.31.18:4824 , piece_id=BRQTN6LIHXSOQNAMIAP5KZC2ZVCK4YV5Y6W2RBSNZLUMAX4I2UWQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.055076449 (632h41m18.926159494s) [noise=true , address=145.79.227.30:4780 , piece_id=K5YFLGQTJVYK5OOU54GMZANYVEFWWVQ3PQ2L4PZUU2WRXR3FPJQQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.055634421 (632h41m18.926717467s) [noise=true , address=64.49.13.26:4954 , piece_id=KBZCJYXHKI2JKZAGP2R52FRRGQLKNEG33ALYLJLALE5JVJEAPGOQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.056155594 (632h41m18.927238629s) [noise=true , address=64.49.13.169:4709 , piece_id=SIK5PLNSLWNBRP36CR4GOSDNWBZ42H67TOPQTXSDWXGFMTTBFD5Q, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.056304037 (632h41m18.927387072s) [noise=true , address=145.79.227.24:4588 , piece_id=DGIL7QT2J72ICIIGAQ4NECOPHPBOAHCITAT6SNHGW76X2XKYYC4A, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.056954675 (632h41m18.92803771s) [noise=true , address=77.90.31.12:4619 , piece_id=2SZZCPGVL77TPTSZCD7YPVJSFVKJGQHYQAMMGOP23VMEELC6DNSA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.057515434 (632h41m18.928598479s) [noise=true , address=145.79.227.14:4864 , piece_id=G4UO6U435XHRMCXQSMAIUVK5SODNPU7FBJZG4ZNJDFVGQWCHPZ3A, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:20.496658364 (632h41m20.367741399s) [upload=554397]: Next returned nil segment Jan 1 00:10:20.496673834 (632h41m20.367756869s) [upload=554397]: Waiting for error group managing segments... Jan 1 00:10:20.806767677 (632h41m20.677850712s) [noise=true , address=64.49.13.140:4511 , piece_id=3HN3335O7DAFTBJA5KQP6UXWZVPZ5VUIB3EFA4NJJWBXRXFJF7JQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:20.947121307 (632h41m20.818204352s) [noise=true , address=145.79.227.28:4760 , piece_id=VELHSBW4LSXCQVAXKZ3XOKF2WNVCL7VUEKB7N6UKKLA5ZPXCVHGA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:20.977079480 (632h41m20.848162525s) [noise=true , address=64.49.13.140:4509 , piece_id=DR2NS5DIRRU4LHOL7VEIAQZLDM6IPBPCCFULLZSFRJOPVUH36DJQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.101757722 (632h41m20.972840757s) [noise=true , address=64.49.13.26:4954 , piece_id=KBZCJYXHKI2JKZAGP2R52FRRGQLKNEG33ALYLJLALE5JVJEAPGOQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.124455562 (632h41m20.995538607s) [noise=true , address=77.90.31.12:4619 , piece_id=2SZZCPGVL77TPTSZCD7YPVJSFVKJGQHYQAMMGOP23VMEELC6DNSA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.166319495 (632h41m21.03740253s) [noise=true , address=77.90.31.11:4581 , piece_id=NNTFVIMH65B65LCG4B33Y5PW3M2ZTBWM5EPJN7LCXFBIW65C4P3Q, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.214239851 (632h41m21.085322886s) [noise=true , address=77.90.31.14:4677 , piece_id=EJOPAXWICBWSBGS4USQCMHX6U4XWECUEPRJJ3RQIPAMHULYOXOFQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.231422007 (632h41m21.102505052s) [noise=true , address=77.90.31.15:4714 , piece_id=Y4TNTQZ2DERVYJ45IWES34IREZM72VWEUJT5DFXBPSFXCXT2ZLBA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.340538969 (632h41m21.211622004s) [noise=true , address=45.91.116.12:4251 , piece_id=S7XHBNCFCHXSC6XEBXTKXGQV2C3QYOZUNJHXUK32MV62OR7THHYA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.642906661 (632h41m21.513989706s) [noise=true , address=145.79.227.14:4864 , piece_id=G4UO6U435XHRMCXQSMAIUVK5SODNPU7FBJZG4ZNJDFVGQWCHPZ3A, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.820646487 (632h41m21.691729532s) [noise=true , address=145.79.227.44:4196 , piece_id=K4EPTOZT3VLMXSZYESGAFHBLVIWA2ULE2VLU37SCJWP7ZMI4LIRQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.856562155 (632h41m21.7276452s) [noise=true , address=145.79.227.30:4780 , piece_id=K5YFLGQTJVYK5OOU54GMZANYVEFWWVQ3PQ2L4PZUU2WRXR3FPJQQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.919986602 (632h41m21.791069647s) [noise=true , address=145.79.227.34:4936 , piece_id=REO4SYQE655A6OWXGR2IARSVCLQ3OS24DDIQJS5ZSL5VO33ZMGFQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:22.039935498 (632h41m21.911018543s) [noise=true , address=77.90.31.18:4824 , piece_id=BRQTN6LIHXSOQNAMIAP5KZC2ZVCK4YV5Y6W2RBSNZLUMAX4I2UWQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:22.039959233 (632h41m21.911042288s) [seg_pos={0 0}, upload=554397]: Segment reached optimal threshold of 14 pieces. Jan 1 00:10:22.040191375 (632h41m21.91127441s) [noise=true , address=145.79.227.33:4877 , piece_id=Z4OYPOCWRMF6L3MDB4TO6I5LPW33SLGZVMLVGFQQOZE2OAE4EYIQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Jan 1 00:10:22.040202396 (632h41m21.911285441s) [noise=true , address=145.79.227.33:4877 , piece_id=Z4OYPOCWRMF6L3MDB4TO6I5LPW33SLGZVMLVGFQQOZE2OAE4EYIQ, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040277219 (632h41m21.911360274s) [noise=true , address=145.79.227.5:4243 , piece_id=RSEPWUQ5TT3NEXUT2YYTNZN5BJQQ24ED2H7DGMV2MMBB3E3TKWCA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FAn2HGJpXP5dfbYfL8PkxCfrBM3SHJVCASQMSy1kaRqLhyiRv): protocol: write tcp 188.166.225.170:53800->145.79.227.5:4243: use of closed network connection; write tcp 188.166.225.170:53800->145.79.227.5:4243: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53800->145.79.227.5:4243: use of closed network connection Jan 1 00:10:22.040288921 (632h41m21.911372176s) [noise=true , address=145.79.227.5:4243 , piece_id=RSEPWUQ5TT3NEXUT2YYTNZN5BJQQ24ED2H7DGMV2MMBB3E3TKWCA, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040324859 (632h41m21.911407894s) [noise=true , address=64.49.13.169:4709 , piece_id=SIK5PLNSLWNBRP36CR4GOSDNWBZ42H67TOPQTXSDWXGFMTTBFD5Q, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:42120->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:42120->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42120->64.49.13.169:4709: use of closed network connection Jan 1 00:10:22.040332203 (632h41m21.911415248s) [noise=true , address=64.49.13.169:4709 , piece_id=SIK5PLNSLWNBRP36CR4GOSDNWBZ42H67TOPQTXSDWXGFMTTBFD5Q, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040454306 (632h41m21.911537341s) [noise=true , address=145.79.227.24:4588 , piece_id=DGIL7QT2J72ICIIGAQ4NECOPHPBOAHCITAT6SNHGW76X2XKYYC4A, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Jan 1 00:10:22.040461780 (632h41m21.911544815s) [noise=true , address=145.79.227.24:4588 , piece_id=DGIL7QT2J72ICIIGAQ4NECOPHPBOAHCITAT6SNHGW76X2XKYYC4A, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040557783 (632h41m21.911640828s) [seg_pos={0 0}, upload=554397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:10:22.040638456 (632h41m21.911721491s) [upload=554397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:10:22.040644648 (632h41m21.911727693s) [upload=554397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:10:22.331369772 (632h41m22.202452807s) [upload=554397]: Done uploading segments... Jan 1 00:10:22.331387687 (632h41m22.202470732s) [upload=554397]: Upload finished. err: <nil> Duration: 33.810577089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X...X............X....X....X......X....X....X.....XXX..X.] duration: 21.547056315s trace: https://jaeger.dev.storj.io/trace/53dd760923edc3ff FAIL (2026-01-01 00:12:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: UWMHSXVKZ6STQZ5LC4RDNOWXRAYTNLA76JZ7D7GUTQCTTN7QRC3Q) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: IE7ERGANAPI2VUVHZXLUULV5LU6E7MUPLSS5BOWUXDONDVIZ5XDQ) context deadline exceeded ======LOGS====== 1 00:11:41.949952472 (632h42m41.821035517s) [upload=554420]: Starting upload Jan 1 00:11:41.950091206 (632h42m41.821174241s) [upload=554420]: Done starting upload Jan 1 00:11:41.950192439 (632h42m41.821275484s) [upload=554420]: Uploading segments... Jan 1 00:11:41.950505234 (632h42m41.821588279s) [upload=554420]: Got next segment. Inline: false Jan 1 00:11:41.950605825 (632h42m41.82168887s) [upload=554420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:11:41.950613130 (632h42m41.821696175s) [upload=554420]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:11:42.195838456 (632h42m42.066921491s) [seg_pos={0 0}, upload=554420]: Begin upload segment... Jan 1 00:11:42.195885456 (632h42m42.066968491s) [seg_pos={0 0}, upload=554420]: Done begin upload segment. Jan 1 00:11:42.195910192 (632h42m42.066993237s) [noise=true , address=45.91.116.6:4976 , piece_id=AKBTGAYVC2NJRTIJX3VCHQTE7SXKTU5SJP7GBK3C6MCJ3PGYXHPQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.196116836 (632h42m42.067199871s) [noise=true , address=77.90.31.14:4663 , piece_id=UWMHSXVKZ6STQZ5LC4RDNOWXRAYTNLA76JZ7D7GUTQCTTN7QRC3Q, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.196537357 (632h42m42.067620392s) [noise=true , address=45.91.116.10:4082 , piece_id=E6ACTYUL4VAYYXCAHG7GMUFAGUXW7NOAJNTXTN2GSQCTTTFCHQEA, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.197151987 (632h42m42.068235032s) [noise=true , address=77.90.31.10:4518 , piece_id=FZR2XRCKGUKH4M3QXY7YG3FFP5LTL4JIM6H4JHGHPOPI4YQLBJYA, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.197636189 (632h42m42.068719224s) [noise=true , address=145.79.227.16:4912 , piece_id=ZG3ZPCTOFITPO53CBT2XYVQ2R2WPKKOQ6DWPNRY6QRGD3RJILYXQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.197654684 (632h42m42.068737729s) [noise=true , address=45.91.116.7:4012 , piece_id=WTMUGF36HNL2XZLD74QFQQ7RYQEKANMOWOVOKDBXF73CABWBD6IQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.198176117 (632h42m42.069259162s) [noise=true , address=145.79.227.38:4978 , piece_id=RGFS23W3GXKIPAM2QEUKPHPA73ZIMAQQPBZAPXAL4GN4REL2ESMQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.198785267 (632h42m42.069868313s) [noise=true , address=45.91.116.13:4154 , piece_id=IE7ERGANAPI2VUVHZXLUULV5LU6E7MUPLSS5BOWUXDONDVIZ5XDQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.199689439 (632h42m42.070772474s) [noise=true , address=145.79.227.19:4077 , piece_id=JYQDFVN3TGJMGARIE4CXWCMNMOZO2BNLB3XGAORIARA4R4N3KGHQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.201012999 (632h42m42.072096034s) [noise=true , address=145.79.227.9:4365 , piece_id=7URIT33NM7EUARFJZCRJDLQHB7SI7PQ7OYL6DQXFJIROJRXZOOWQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.201412670 (632h42m42.072495705s) [noise=true , address=145.79.227.11:4730 , piece_id=ZM23S4LC4C6H3PZC4WHBK7R57MCZZSFNAI6R6IM3UTGUVRTRD6IQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.201817722 (632h42m42.072900757s) [noise=true , address=145.79.227.3:4151 , piece_id=FCT37VLJQ3K334JGRHAZTL6IEFLBXCJ6E4AD6ANIA2YJVCSPIMSQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.202401594 (632h42m42.073484629s) [noise=true , address=145.79.227.15:4873 , piece_id=6XSMNC5455BL7LXNEMWDZAWD4LNTYNYU3VGL56ZBGRM77LIJ6DDQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.202856600 (632h42m42.073939645s) [noise=true , address=145.79.227.50:4373 , piece_id=6BUP2CXGVY3KSDMG2ZZF4QNNAIHSSGBFL72T4L2OUNH4EHRDLOYA, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.204577476 (632h42m42.075660521s) [noise=true , address=145.79.227.24:4583 , piece_id=JRMHJEBQEDULMLRANAZUQQ544UG5OHXNJGAO56R6TGF75SVCUNYQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.207128464 (632h42m42.078211499s) [noise=true , address=145.79.227.10:4401 , piece_id=M53H7CHT2NDVXPE7HRZIUNF64BH46YSVKDLK2EKE2W6UQR7M2KNQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:43.894621269 (632h42m43.765704304s) [upload=554420]: Next returned nil segment Jan 1 00:11:43.894641658 (632h42m43.765724704s) [upload=554420]: Waiting for error group managing segments... Jan 1 00:11:44.177169193 (632h42m44.048252238s) [noise=true , address=145.79.227.11:4730 , piece_id=ZM23S4LC4C6H3PZC4WHBK7R57MCZZSFNAI6R6IM3UTGUVRTRD6IQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.177604331 (632h42m44.048687366s) [noise=true , address=145.79.227.50:4373 , piece_id=6BUP2CXGVY3KSDMG2ZZF4QNNAIHSSGBFL72T4L2OUNH4EHRDLOYA, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.191346609 (632h42m44.062429644s) [noise=true , address=145.79.227.19:4077 , piece_id=JYQDFVN3TGJMGARIE4CXWCMNMOZO2BNLB3XGAORIARA4R4N3KGHQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.193352708 (632h42m44.064435753s) [noise=true , address=145.79.227.3:4151 , piece_id=FCT37VLJQ3K334JGRHAZTL6IEFLBXCJ6E4AD6ANIA2YJVCSPIMSQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.193746588 (632h42m44.064829623s) [noise=true , address=145.79.227.15:4873 , piece_id=6XSMNC5455BL7LXNEMWDZAWD4LNTYNYU3VGL56ZBGRM77LIJ6DDQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.198818185 (632h42m44.06990123s) [noise=true , address=145.79.227.9:4365 , piece_id=7URIT33NM7EUARFJZCRJDLQHB7SI7PQ7OYL6DQXFJIROJRXZOOWQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.211799272 (632h42m44.082882318s) [noise=true , address=145.79.227.38:4978 , piece_id=RGFS23W3GXKIPAM2QEUKPHPA73ZIMAQQPBZAPXAL4GN4REL2ESMQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.217579309 (632h42m44.088662354s) [noise=true , address=145.79.227.16:4912 , piece_id=ZG3ZPCTOFITPO53CBT2XYVQ2R2WPKKOQ6DWPNRY6QRGD3RJILYXQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.218513196 (632h42m44.089596261s) [noise=true , address=145.79.227.24:4583 , piece_id=JRMHJEBQEDULMLRANAZUQQ544UG5OHXNJGAO56R6TGF75SVCUNYQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.220628164 (632h42m44.091711209s) [noise=true , address=145.79.227.10:4401 , piece_id=M53H7CHT2NDVXPE7HRZIUNF64BH46YSVKDLK2EKE2W6UQR7M2KNQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.440555750 (632h42m44.311638785s) [noise=true , address=77.90.31.14:4663 , piece_id=UWMHSXVKZ6STQZ5LC4RDNOWXRAYTNLA76JZ7D7GUTQCTTN7QRC3Q, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.501467963 (632h42m44.372550998s) [noise=true , address=45.91.116.13:4154 , piece_id=IE7ERGANAPI2VUVHZXLUULV5LU6E7MUPLSS5BOWUXDONDVIZ5XDQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.588674244 (632h42m44.459757289s) [noise=true , address=45.91.116.7:4012 , piece_id=WTMUGF36HNL2XZLD74QFQQ7RYQEKANMOWOVOKDBXF73CABWBD6IQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.588696677 (632h42m44.459779722s) [seg_pos={0 0}, upload=554420]: Segment reached optimal threshold of 13 pieces. Jan 1 00:11:44.588904032 (632h42m44.459987067s) [noise=true , address=45.91.116.6:4976 , piece_id=AKBTGAYVC2NJRTIJX3VCHQTE7SXKTU5SJP7GBK3C6MCJ3PGYXHPQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 00:11:44.588911256 (632h42m44.459994291s) [noise=true , address=45.91.116.6:4976 , piece_id=AKBTGAYVC2NJRTIJX3VCHQTE7SXKTU5SJP7GBK3C6MCJ3PGYXHPQ, seg_pos={0 0}, upload=554420]: Long tail cancellation due to optimal threshold Jan 1 00:11:44.588962394 (632h42m44.460045439s) [noise=true , address=77.90.31.10:4518 , piece_id=FZR2XRCKGUKH4M3QXY7YG3FFP5LTL4JIM6H4JHGHPOPI4YQLBJYA, seg_pos={0 0}, upload=554420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:34074->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:34074->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34074->77.90.31.10:4518: use of closed network connection Jan 1 00:11:44.588972573 (632h42m44.460055618s) [noise=true , address=77.90.31.10:4518 , piece_id=FZR2XRCKGUKH4M3QXY7YG3FFP5LTL4JIM6H4JHGHPOPI4YQLBJYA, seg_pos={0 0}, upload=554420]: Long tail cancellation due to optimal threshold Jan 1 00:11:44.589162293 (632h42m44.460245328s) [noise=true , address=45.91.116.10:4082 , piece_id=E6ACTYUL4VAYYXCAHG7GMUFAGUXW7NOAJNTXTN2GSQCTTTFCHQEA, seg_pos={0 0}, upload=554420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 1 00:11:44.589169838 (632h42m44.460252883s) [noise=true , address=45.91.116.10:4082 , piece_id=E6ACTYUL4VAYYXCAHG7GMUFAGUXW7NOAJNTXTN2GSQCTTTFCHQEA, seg_pos={0 0}, upload=554420]: Long tail cancellation due to optimal threshold Jan 1 00:11:44.589187091 (632h42m44.460270136s) [seg_pos={0 0}, upload=554420]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:11:44.589266151 (632h42m44.460349206s) [upload=554420]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:11:44.589271551 (632h42m44.460354586s) [upload=554420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:11:44.892025329 (632h42m44.763108364s) [upload=554420]: Done uploading segments... Jan 1 00:11:44.892050196 (632h42m44.763133241s) [upload=554420]: Upload finished. err: <nil> Duration: 21.547240925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X...X............X....X....X......X....X....X.....XXX..X] duration: 30.007031286s trace: https://jaeger.dev.storj.io/trace/53dd760923edc3ff FAIL (2026-01-01 00:12:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: UWMHSXVKZ6STQZ5LC4RDNOWXRAYTNLA76JZ7D7GUTQCTTN7QRC3Q) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: IE7ERGANAPI2VUVHZXLUULV5LU6E7MUPLSS5BOWUXDONDVIZ5XDQ) context deadline exceeded ======LOGS====== 1 00:11:41.949952472 (632h42m41.821035517s) [upload=554420]: Starting upload Jan 1 00:11:41.950091206 (632h42m41.821174241s) [upload=554420]: Done starting upload Jan 1 00:11:41.950192439 (632h42m41.821275484s) [upload=554420]: Uploading segments... Jan 1 00:11:41.950505234 (632h42m41.821588279s) [upload=554420]: Got next segment. Inline: false Jan 1 00:11:41.950605825 (632h42m41.82168887s) [upload=554420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:11:41.950613130 (632h42m41.821696175s) [upload=554420]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:11:42.195838456 (632h42m42.066921491s) [seg_pos={0 0}, upload=554420]: Begin upload segment... Jan 1 00:11:42.195885456 (632h42m42.066968491s) [seg_pos={0 0}, upload=554420]: Done begin upload segment. Jan 1 00:11:42.195910192 (632h42m42.066993237s) [noise=true , address=45.91.116.6:4976 , piece_id=AKBTGAYVC2NJRTIJX3VCHQTE7SXKTU5SJP7GBK3C6MCJ3PGYXHPQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.196116836 (632h42m42.067199871s) [noise=true , address=77.90.31.14:4663 , piece_id=UWMHSXVKZ6STQZ5LC4RDNOWXRAYTNLA76JZ7D7GUTQCTTN7QRC3Q, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.196537357 (632h42m42.067620392s) [noise=true , address=45.91.116.10:4082 , piece_id=E6ACTYUL4VAYYXCAHG7GMUFAGUXW7NOAJNTXTN2GSQCTTTFCHQEA, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.197151987 (632h42m42.068235032s) [noise=true , address=77.90.31.10:4518 , piece_id=FZR2XRCKGUKH4M3QXY7YG3FFP5LTL4JIM6H4JHGHPOPI4YQLBJYA, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.197636189 (632h42m42.068719224s) [noise=true , address=145.79.227.16:4912 , piece_id=ZG3ZPCTOFITPO53CBT2XYVQ2R2WPKKOQ6DWPNRY6QRGD3RJILYXQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.197654684 (632h42m42.068737729s) [noise=true , address=45.91.116.7:4012 , piece_id=WTMUGF36HNL2XZLD74QFQQ7RYQEKANMOWOVOKDBXF73CABWBD6IQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.198176117 (632h42m42.069259162s) [noise=true , address=145.79.227.38:4978 , piece_id=RGFS23W3GXKIPAM2QEUKPHPA73ZIMAQQPBZAPXAL4GN4REL2ESMQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.198785267 (632h42m42.069868313s) [noise=true , address=45.91.116.13:4154 , piece_id=IE7ERGANAPI2VUVHZXLUULV5LU6E7MUPLSS5BOWUXDONDVIZ5XDQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.199689439 (632h42m42.070772474s) [noise=true , address=145.79.227.19:4077 , piece_id=JYQDFVN3TGJMGARIE4CXWCMNMOZO2BNLB3XGAORIARA4R4N3KGHQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.201012999 (632h42m42.072096034s) [noise=true , address=145.79.227.9:4365 , piece_id=7URIT33NM7EUARFJZCRJDLQHB7SI7PQ7OYL6DQXFJIROJRXZOOWQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.201412670 (632h42m42.072495705s) [noise=true , address=145.79.227.11:4730 , piece_id=ZM23S4LC4C6H3PZC4WHBK7R57MCZZSFNAI6R6IM3UTGUVRTRD6IQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.201817722 (632h42m42.072900757s) [noise=true , address=145.79.227.3:4151 , piece_id=FCT37VLJQ3K334JGRHAZTL6IEFLBXCJ6E4AD6ANIA2YJVCSPIMSQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.202401594 (632h42m42.073484629s) [noise=true , address=145.79.227.15:4873 , piece_id=6XSMNC5455BL7LXNEMWDZAWD4LNTYNYU3VGL56ZBGRM77LIJ6DDQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.202856600 (632h42m42.073939645s) [noise=true , address=145.79.227.50:4373 , piece_id=6BUP2CXGVY3KSDMG2ZZF4QNNAIHSSGBFL72T4L2OUNH4EHRDLOYA, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.204577476 (632h42m42.075660521s) [noise=true , address=145.79.227.24:4583 , piece_id=JRMHJEBQEDULMLRANAZUQQ544UG5OHXNJGAO56R6TGF75SVCUNYQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:42.207128464 (632h42m42.078211499s) [noise=true , address=145.79.227.10:4401 , piece_id=M53H7CHT2NDVXPE7HRZIUNF64BH46YSVKDLK2EKE2W6UQR7M2KNQ, seg_pos={0 0}, upload=554420]: Uploading piece... Jan 1 00:11:43.894621269 (632h42m43.765704304s) [upload=554420]: Next returned nil segment Jan 1 00:11:43.894641658 (632h42m43.765724704s) [upload=554420]: Waiting for error group managing segments... Jan 1 00:11:44.177169193 (632h42m44.048252238s) [noise=true , address=145.79.227.11:4730 , piece_id=ZM23S4LC4C6H3PZC4WHBK7R57MCZZSFNAI6R6IM3UTGUVRTRD6IQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.177604331 (632h42m44.048687366s) [noise=true , address=145.79.227.50:4373 , piece_id=6BUP2CXGVY3KSDMG2ZZF4QNNAIHSSGBFL72T4L2OUNH4EHRDLOYA, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.191346609 (632h42m44.062429644s) [noise=true , address=145.79.227.19:4077 , piece_id=JYQDFVN3TGJMGARIE4CXWCMNMOZO2BNLB3XGAORIARA4R4N3KGHQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.193352708 (632h42m44.064435753s) [noise=true , address=145.79.227.3:4151 , piece_id=FCT37VLJQ3K334JGRHAZTL6IEFLBXCJ6E4AD6ANIA2YJVCSPIMSQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.193746588 (632h42m44.064829623s) [noise=true , address=145.79.227.15:4873 , piece_id=6XSMNC5455BL7LXNEMWDZAWD4LNTYNYU3VGL56ZBGRM77LIJ6DDQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.198818185 (632h42m44.06990123s) [noise=true , address=145.79.227.9:4365 , piece_id=7URIT33NM7EUARFJZCRJDLQHB7SI7PQ7OYL6DQXFJIROJRXZOOWQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.211799272 (632h42m44.082882318s) [noise=true , address=145.79.227.38:4978 , piece_id=RGFS23W3GXKIPAM2QEUKPHPA73ZIMAQQPBZAPXAL4GN4REL2ESMQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.217579309 (632h42m44.088662354s) [noise=true , address=145.79.227.16:4912 , piece_id=ZG3ZPCTOFITPO53CBT2XYVQ2R2WPKKOQ6DWPNRY6QRGD3RJILYXQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.218513196 (632h42m44.089596261s) [noise=true , address=145.79.227.24:4583 , piece_id=JRMHJEBQEDULMLRANAZUQQ544UG5OHXNJGAO56R6TGF75SVCUNYQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.220628164 (632h42m44.091711209s) [noise=true , address=145.79.227.10:4401 , piece_id=M53H7CHT2NDVXPE7HRZIUNF64BH46YSVKDLK2EKE2W6UQR7M2KNQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.440555750 (632h42m44.311638785s) [noise=true , address=77.90.31.14:4663 , piece_id=UWMHSXVKZ6STQZ5LC4RDNOWXRAYTNLA76JZ7D7GUTQCTTN7QRC3Q, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.501467963 (632h42m44.372550998s) [noise=true , address=45.91.116.13:4154 , piece_id=IE7ERGANAPI2VUVHZXLUULV5LU6E7MUPLSS5BOWUXDONDVIZ5XDQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.588674244 (632h42m44.459757289s) [noise=true , address=45.91.116.7:4012 , piece_id=WTMUGF36HNL2XZLD74QFQQ7RYQEKANMOWOVOKDBXF73CABWBD6IQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: <nil> Jan 1 00:11:44.588696677 (632h42m44.459779722s) [seg_pos={0 0}, upload=554420]: Segment reached optimal threshold of 13 pieces. Jan 1 00:11:44.588904032 (632h42m44.459987067s) [noise=true , address=45.91.116.6:4976 , piece_id=AKBTGAYVC2NJRTIJX3VCHQTE7SXKTU5SJP7GBK3C6MCJ3PGYXHPQ, seg_pos={0 0}, upload=554420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 00:11:44.588911256 (632h42m44.459994291s) [noise=true , address=45.91.116.6:4976 , piece_id=AKBTGAYVC2NJRTIJX3VCHQTE7SXKTU5SJP7GBK3C6MCJ3PGYXHPQ, seg_pos={0 0}, upload=554420]: Long tail cancellation due to optimal threshold Jan 1 00:11:44.588962394 (632h42m44.460045439s) [noise=true , address=77.90.31.10:4518 , piece_id=FZR2XRCKGUKH4M3QXY7YG3FFP5LTL4JIM6H4JHGHPOPI4YQLBJYA, seg_pos={0 0}, upload=554420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:34074->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:34074->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34074->77.90.31.10:4518: use of closed network connection Jan 1 00:11:44.588972573 (632h42m44.460055618s) [noise=true , address=77.90.31.10:4518 , piece_id=FZR2XRCKGUKH4M3QXY7YG3FFP5LTL4JIM6H4JHGHPOPI4YQLBJYA, seg_pos={0 0}, upload=554420]: Long tail cancellation due to optimal threshold Jan 1 00:11:44.589162293 (632h42m44.460245328s) [noise=true , address=45.91.116.10:4082 , piece_id=E6ACTYUL4VAYYXCAHG7GMUFAGUXW7NOAJNTXTN2GSQCTTTFCHQEA, seg_pos={0 0}, upload=554420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 1 00:11:44.589169838 (632h42m44.460252883s) [noise=true , address=45.91.116.10:4082 , piece_id=E6ACTYUL4VAYYXCAHG7GMUFAGUXW7NOAJNTXTN2GSQCTTTFCHQEA, seg_pos={0 0}, upload=554420]: Long tail cancellation due to optimal threshold Jan 1 00:11:44.589187091 (632h42m44.460270136s) [seg_pos={0 0}, upload=554420]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:11:44.589266151 (632h42m44.460349206s) [upload=554420]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:11:44.589271551 (632h42m44.460354586s) [upload=554420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:11:44.892025329 (632h42m44.763108364s) [upload=554420]: Done uploading segments... Jan 1 00:11:44.892050196 (632h42m44.763133241s) [upload=554420]: Upload finished. err: <nil> Duration: 30.007054629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....................X...........X.] duration: 30.29570333s trace: https://jaeger.dev.storj.io/trace/67e2279657e45584 FAIL (2026-01-01 00:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, Piece ID: Y4TNTQZ2DERVYJ45IWES34IREZM72VWEUJT5DFXBPSFXCXT2ZLBA) context deadline exceeded; piecestore: (Node ID: 12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, Piece ID: 2SZZCPGVL77TPTSZCD7YPVJSFVKJGQHYQAMMGOP23VMEELC6DNSA) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: EJOPAXWICBWSBGS4USQCMHX6U4XWECUEPRJJ3RQIPAMHULYOXOFQ) context deadline exceeded; piecestore: (Node ID: 12ZNCRKd7xMFNHfCJBJ38yfyUmczhYwJetKtiQxX9SDrwxRhzTU, Piece ID: S7XHBNCFCHXSC6XEBXTKXGQV2C3QYOZUNJHXUK32MV62OR7THHYA) context deadline exceeded ======LOGS====== :4936 , piece_id=REO4SYQE655A6OWXGR2IARSVCLQ3OS24DDIQJS5ZSL5VO33ZMGFQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.050418220 (632h41m18.921501255s) [noise=true , address=77.90.31.14:4677 , piece_id=EJOPAXWICBWSBGS4USQCMHX6U4XWECUEPRJJ3RQIPAMHULYOXOFQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.050930676 (632h41m18.922013721s) [noise=true , address=77.90.31.15:4714 , piece_id=Y4TNTQZ2DERVYJ45IWES34IREZM72VWEUJT5DFXBPSFXCXT2ZLBA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.051454794 (632h41m18.922537839s) [noise=true , address=45.91.116.12:4251 , piece_id=S7XHBNCFCHXSC6XEBXTKXGQV2C3QYOZUNJHXUK32MV62OR7THHYA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.051881115 (632h41m18.92296416s) [noise=true , address=64.49.13.140:4509 , piece_id=DR2NS5DIRRU4LHOL7VEIAQZLDM6IPBPCCFULLZSFRJOPVUH36DJQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.052297879 (632h41m18.923380924s) [noise=true , address=64.49.13.140:4511 , piece_id=3HN3335O7DAFTBJA5KQP6UXWZVPZ5VUIB3EFA4NJJWBXRXFJF7JQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.052727466 (632h41m18.923810502s) [noise=true , address=77.90.31.11:4581 , piece_id=NNTFVIMH65B65LCG4B33Y5PW3M2ZTBWM5EPJN7LCXFBIW65C4P3Q, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.053205347 (632h41m18.924288392s) [noise=true , address=145.79.227.28:4760 , piece_id=VELHSBW4LSXCQVAXKZ3XOKF2WNVCL7VUEKB7N6UKKLA5ZPXCVHGA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.054372899 (632h41m18.925455954s) [noise=true , address=77.90.31.18:4824 , piece_id=BRQTN6LIHXSOQNAMIAP5KZC2ZVCK4YV5Y6W2RBSNZLUMAX4I2UWQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.055076449 (632h41m18.926159494s) [noise=true , address=145.79.227.30:4780 , piece_id=K5YFLGQTJVYK5OOU54GMZANYVEFWWVQ3PQ2L4PZUU2WRXR3FPJQQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.055634421 (632h41m18.926717467s) [noise=true , address=64.49.13.26:4954 , piece_id=KBZCJYXHKI2JKZAGP2R52FRRGQLKNEG33ALYLJLALE5JVJEAPGOQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.056155594 (632h41m18.927238629s) [noise=true , address=64.49.13.169:4709 , piece_id=SIK5PLNSLWNBRP36CR4GOSDNWBZ42H67TOPQTXSDWXGFMTTBFD5Q, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.056304037 (632h41m18.927387072s) [noise=true , address=145.79.227.24:4588 , piece_id=DGIL7QT2J72ICIIGAQ4NECOPHPBOAHCITAT6SNHGW76X2XKYYC4A, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.056954675 (632h41m18.92803771s) [noise=true , address=77.90.31.12:4619 , piece_id=2SZZCPGVL77TPTSZCD7YPVJSFVKJGQHYQAMMGOP23VMEELC6DNSA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.057515434 (632h41m18.928598479s) [noise=true , address=145.79.227.14:4864 , piece_id=G4UO6U435XHRMCXQSMAIUVK5SODNPU7FBJZG4ZNJDFVGQWCHPZ3A, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:20.496658364 (632h41m20.367741399s) [upload=554397]: Next returned nil segment Jan 1 00:10:20.496673834 (632h41m20.367756869s) [upload=554397]: Waiting for error group managing segments... Jan 1 00:10:20.806767677 (632h41m20.677850712s) [noise=true , address=64.49.13.140:4511 , piece_id=3HN3335O7DAFTBJA5KQP6UXWZVPZ5VUIB3EFA4NJJWBXRXFJF7JQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:20.947121307 (632h41m20.818204352s) [noise=true , address=145.79.227.28:4760 , piece_id=VELHSBW4LSXCQVAXKZ3XOKF2WNVCL7VUEKB7N6UKKLA5ZPXCVHGA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:20.977079480 (632h41m20.848162525s) [noise=true , address=64.49.13.140:4509 , piece_id=DR2NS5DIRRU4LHOL7VEIAQZLDM6IPBPCCFULLZSFRJOPVUH36DJQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.101757722 (632h41m20.972840757s) [noise=true , address=64.49.13.26:4954 , piece_id=KBZCJYXHKI2JKZAGP2R52FRRGQLKNEG33ALYLJLALE5JVJEAPGOQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.124455562 (632h41m20.995538607s) [noise=true , address=77.90.31.12:4619 , piece_id=2SZZCPGVL77TPTSZCD7YPVJSFVKJGQHYQAMMGOP23VMEELC6DNSA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.166319495 (632h41m21.03740253s) [noise=true , address=77.90.31.11:4581 , piece_id=NNTFVIMH65B65LCG4B33Y5PW3M2ZTBWM5EPJN7LCXFBIW65C4P3Q, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.214239851 (632h41m21.085322886s) [noise=true , address=77.90.31.14:4677 , piece_id=EJOPAXWICBWSBGS4USQCMHX6U4XWECUEPRJJ3RQIPAMHULYOXOFQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.231422007 (632h41m21.102505052s) [noise=true , address=77.90.31.15:4714 , piece_id=Y4TNTQZ2DERVYJ45IWES34IREZM72VWEUJT5DFXBPSFXCXT2ZLBA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.340538969 (632h41m21.211622004s) [noise=true , address=45.91.116.12:4251 , piece_id=S7XHBNCFCHXSC6XEBXTKXGQV2C3QYOZUNJHXUK32MV62OR7THHYA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.642906661 (632h41m21.513989706s) [noise=true , address=145.79.227.14:4864 , piece_id=G4UO6U435XHRMCXQSMAIUVK5SODNPU7FBJZG4ZNJDFVGQWCHPZ3A, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.820646487 (632h41m21.691729532s) [noise=true , address=145.79.227.44:4196 , piece_id=K4EPTOZT3VLMXSZYESGAFHBLVIWA2ULE2VLU37SCJWP7ZMI4LIRQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.856562155 (632h41m21.7276452s) [noise=true , address=145.79.227.30:4780 , piece_id=K5YFLGQTJVYK5OOU54GMZANYVEFWWVQ3PQ2L4PZUU2WRXR3FPJQQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.919986602 (632h41m21.791069647s) [noise=true , address=145.79.227.34:4936 , piece_id=REO4SYQE655A6OWXGR2IARSVCLQ3OS24DDIQJS5ZSL5VO33ZMGFQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:22.039935498 (632h41m21.911018543s) [noise=true , address=77.90.31.18:4824 , piece_id=BRQTN6LIHXSOQNAMIAP5KZC2ZVCK4YV5Y6W2RBSNZLUMAX4I2UWQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:22.039959233 (632h41m21.911042288s) [seg_pos={0 0}, upload=554397]: Segment reached optimal threshold of 14 pieces. Jan 1 00:10:22.040191375 (632h41m21.91127441s) [noise=true , address=145.79.227.33:4877 , piece_id=Z4OYPOCWRMF6L3MDB4TO6I5LPW33SLGZVMLVGFQQOZE2OAE4EYIQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Jan 1 00:10:22.040202396 (632h41m21.911285441s) [noise=true , address=145.79.227.33:4877 , piece_id=Z4OYPOCWRMF6L3MDB4TO6I5LPW33SLGZVMLVGFQQOZE2OAE4EYIQ, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040277219 (632h41m21.911360274s) [noise=true , address=145.79.227.5:4243 , piece_id=RSEPWUQ5TT3NEXUT2YYTNZN5BJQQ24ED2H7DGMV2MMBB3E3TKWCA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FAn2HGJpXP5dfbYfL8PkxCfrBM3SHJVCASQMSy1kaRqLhyiRv): protocol: write tcp 188.166.225.170:53800->145.79.227.5:4243: use of closed network connection; write tcp 188.166.225.170:53800->145.79.227.5:4243: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53800->145.79.227.5:4243: use of closed network connection Jan 1 00:10:22.040288921 (632h41m21.911372176s) [noise=true , address=145.79.227.5:4243 , piece_id=RSEPWUQ5TT3NEXUT2YYTNZN5BJQQ24ED2H7DGMV2MMBB3E3TKWCA, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040324859 (632h41m21.911407894s) [noise=true , address=64.49.13.169:4709 , piece_id=SIK5PLNSLWNBRP36CR4GOSDNWBZ42H67TOPQTXSDWXGFMTTBFD5Q, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:42120->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:42120->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42120->64.49.13.169:4709: use of closed network connection Jan 1 00:10:22.040332203 (632h41m21.911415248s) [noise=true , address=64.49.13.169:4709 , piece_id=SIK5PLNSLWNBRP36CR4GOSDNWBZ42H67TOPQTXSDWXGFMTTBFD5Q, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040454306 (632h41m21.911537341s) [noise=true , address=145.79.227.24:4588 , piece_id=DGIL7QT2J72ICIIGAQ4NECOPHPBOAHCITAT6SNHGW76X2XKYYC4A, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Jan 1 00:10:22.040461780 (632h41m21.911544815s) [noise=true , address=145.79.227.24:4588 , piece_id=DGIL7QT2J72ICIIGAQ4NECOPHPBOAHCITAT6SNHGW76X2XKYYC4A, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040557783 (632h41m21.911640828s) [seg_pos={0 0}, upload=554397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:10:22.040638456 (632h41m21.911721491s) [upload=554397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:10:22.040644648 (632h41m21.911727693s) [upload=554397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:10:22.331369772 (632h41m22.202452807s) [upload=554397]: Done uploading segments... Jan 1 00:10:22.331387687 (632h41m22.202470732s) [upload=554397]: Upload finished. err: <nil> Duration: 30.295720793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X....X...X............X....X....X......X....X....X.....XXX..] duration: 19.2598934s trace: https://jaeger.dev.storj.io/trace/23758042101091ca FAIL (2026-01-01 00:11:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 00:09:45.465120922 (632h40m45.336203977s) [upload=554390]: Starting upload Jan 1 00:09:45.465208038 (632h40m45.336291083s) [upload=554390]: Done starting upload Jan 1 00:09:45.465279364 (632h40m45.336362409s) [upload=554390]: Uploading segments... Jan 1 00:09:45.465460078 (632h40m45.336543123s) [upload=554390]: Got next segment. Inline: false Jan 1 00:09:45.465539890 (632h40m45.336622935s) [upload=554390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:09:45.465550621 (632h40m45.336633666s) [upload=554390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:09:45.715683168 (632h40m45.586766203s) [seg_pos={0 0}, upload=554390]: Begin upload segment... Jan 1 00:09:45.715733694 (632h40m45.586816729s) [seg_pos={0 0}, upload=554390]: Done begin upload segment. Jan 1 00:09:45.715765865 (632h40m45.5868489s) [noise=true , address=145.79.227.44:4194 , piece_id=VY4CCXV3YXPGQTZC3N2SS5V5ZAT43HAR3CEPU2GSNKZH2TUPRHVA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.715781495 (632h40m45.58686453s) [noise=true , address=145.79.227.36:4519 , piece_id=CBBMCLRSIGUWSXADVOV2EQEC7RCIZN7JSQTJAFJADMWVXWEVJ2JA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.716265758 (632h40m45.587348793s) [noise=true , address=145.79.227.42:4123 , piece_id=SHPQ3LZ46F5MLCAP6574MXPXCWOYHVD2CGWW7OXYMWQ2D2TCEJ4Q, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.716650530 (632h40m45.587733565s) [noise=true , address=145.79.227.4:4187 , piece_id=TFHN6TIXVMM2OWBS3IA3624UEUB6WMNRQILG3JVBTDCMX5XH4VEA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717004985 (632h40m45.58808802s) [noise=true , address=45.91.116.7:4012 , piece_id=EVWNGG3YM7I23OIRI46V7VUGTV45G2EVBLJS3QBBB3YJYPIG3W4A, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717333260 (632h40m45.588416305s) [noise=true , address=145.79.227.26:4655 , piece_id=IZSEUCFPZZZTUA53C27DAE7DUTYUZDOFD3QYKYZWXH6IOW7NBB6Q, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717573066 (632h40m45.588656111s) [noise=true , address=77.90.31.17:4770 , piece_id=NRFQJP7ZYK6OY73AVLRJT3BSLQNHZJFQPJEID46D6FJ47LUH4AEA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717926149 (632h40m45.589009184s) [noise=true , address=45.91.116.13:4154 , piece_id=U4HM6NKCSACXDKIVCFNDBBWSPAR3XBXSQXGUC7S7KMOZ4K3F6WAQ, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718271356 (632h40m45.589354391s) [noise=true , address=145.79.227.11:4730 , piece_id=CMMVQOCGO7BLPGOL5MGZTTLICEOOPVRF4OYDRYQAPBIJPC5GHVXQ, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718453132 (632h40m45.589536167s) [noise=true , address=77.90.31.12:4590 , piece_id=DXF6AZXWRRGFCR7L2O4BB67W2IIK6AYOZGL4YDXH27EQF35GIAGA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718639016 (632h40m45.589722052s) [noise=true , address=145.79.227.6:4259 , piece_id=GNSHMUVXCYOOM4JVTPDJJA4VUZJDQPPMDBQW6F5R62WL3WHVWPDQ, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718954607 (632h40m45.590037652s) [noise=true , address=77.90.31.15:4700 , piece_id=XUBRSQ6NXMJ44NQOGXSRIFAWRLPE73YURXV5LMVZAZPL3EQPQSPA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718960418 (632h40m45.590043463s) [noise=true , address=77.90.31.21:4880 , piece_id=MTSDTM75BWGDVUU2BNR6N3SR5S4TJALRQU6PTNW4XGPSI6Q24FIA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.719254398 (632h40m45.590337433s) [noise=true , address=145.79.227.17:4972 , piece_id=FBZO5BZ3LOTOBL7WJUI4KILPWZIF4WISNDK4HLZX6PNAOSOHN6PA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.719346163 (632h40m45.590429198s) [noise=true , address=45.91.116.4:4937 , piece_id=S76Z4ZQLXAZCBAYSUB6JNZXHJ3HSZJEIHWRR24HDN34QDM6WGIWA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.719546875 (632h40m45.59062992s) [noise=true , address=45.91.116.10:4084 , piece_id=J6CCFXCKYZGJBGTX6H7VGC3JYPG5HUQ5JX666VTVDY4NJ3BG444Q, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:47.414914601 (632h40m47.285997636s) [upload=554390]: Next returned nil segment Jan 1 00:09:47.414928316 (632h40m47.286011352s) [upload=554390]: Waiting for error group managing segments... Jan 1 00:09:47.696668918 (632h40m47.567751953s) [noise=true , address=145.79.227.4:4187 , piece_id=TFHN6TIXVMM2OWBS3IA3624UEUB6WMNRQILG3JVBTDCMX5XH4VEA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.700534478 (632h40m47.571617523s) [noise=true , address=145.79.227.11:4730 , piece_id=CMMVQOCGO7BLPGOL5MGZTTLICEOOPVRF4OYDRYQAPBIJPC5GHVXQ, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.703399113 (632h40m47.574482148s) [noise=true , address=145.79.227.17:4972 , piece_id=FBZO5BZ3LOTOBL7WJUI4KILPWZIF4WISNDK4HLZX6PNAOSOHN6PA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.712386226 (632h40m47.583469271s) [noise=true , address=145.79.227.42:4123 , piece_id=SHPQ3LZ46F5MLCAP6574MXPXCWOYHVD2CGWW7OXYMWQ2D2TCEJ4Q, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.720236954 (632h40m47.591319989s) [noise=true , address=145.79.227.36:4519 , piece_id=CBBMCLRSIGUWSXADVOV2EQEC7RCIZN7JSQTJAFJADMWVXWEVJ2JA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.735226475 (632h40m47.60630951s) [noise=true , address=145.79.227.6:4259 , piece_id=GNSHMUVXCYOOM4JVTPDJJA4VUZJDQPPMDBQW6F5R62WL3WHVWPDQ, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.737387791 (632h40m47.608470826s) [noise=true , address=145.79.227.26:4655 , piece_id=IZSEUCFPZZZTUA53C27DAE7DUTYUZDOFD3QYKYZWXH6IOW7NBB6Q, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.885902698 (632h40m47.756985734s) [noise=true , address=77.90.31.12:4590 , piece_id=DXF6AZXWRRGFCR7L2O4BB67W2IIK6AYOZGL4YDXH27EQF35GIAGA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.994421789 (632h40m47.865504824s) [noise=true , address=45.91.116.13:4154 , piece_id=U4HM6NKCSACXDKIVCFNDBBWSPAR3XBXSQXGUC7S7KMOZ4K3F6WAQ, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.004618224 (632h40m47.875701269s) [noise=true , address=77.90.31.15:4700 , piece_id=XUBRSQ6NXMJ44NQOGXSRIFAWRLPE73YURXV5LMVZAZPL3EQPQSPA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.071212837 (632h40m47.942295882s) [noise=true , address=45.91.116.4:4937 , piece_id=S76Z4ZQLXAZCBAYSUB6JNZXHJ3HSZJEIHWRR24HDN34QDM6WGIWA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.077205898 (632h40m47.948288933s) [noise=true , address=45.91.116.10:4084 , piece_id=J6CCFXCKYZGJBGTX6H7VGC3JYPG5HUQ5JX666VTVDY4NJ3BG444Q, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.730471781 (632h40m48.601554816s) [noise=true , address=45.91.116.7:4012 , piece_id=EVWNGG3YM7I23OIRI46V7VUGTV45G2EVBLJS3QBBB3YJYPIG3W4A, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.730493042 (632h40m48.601576077s) [seg_pos={0 0}, upload=554390]: Segment reached optimal threshold of 13 pieces. Jan 1 00:09:48.730733249 (632h40m48.601816294s) [noise=true , address=77.90.31.21:4880 , piece_id=MTSDTM75BWGDVUU2BNR6N3SR5S4TJALRQU6PTNW4XGPSI6Q24FIA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Jan 1 00:09:48.730739471 (632h40m48.601822506s) [noise=true , address=77.90.31.21:4880 , piece_id=MTSDTM75BWGDVUU2BNR6N3SR5S4TJALRQU6PTNW4XGPSI6Q24FIA, seg_pos={0 0}, upload=554390]: Long tail cancellation due to optimal threshold Jan 1 00:09:48.730936556 (632h40m48.602019601s) [noise=true , address=77.90.31.17:4770 , piece_id=NRFQJP7ZYK6OY73AVLRJT3BSLQNHZJFQPJEID46D6FJ47LUH4AEA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 1 00:09:48.730943088 (632h40m48.602026133s) [noise=true , address=77.90.31.17:4770 , piece_id=NRFQJP7ZYK6OY73AVLRJT3BSLQNHZJFQPJEID46D6FJ47LUH4AEA, seg_pos={0 0}, upload=554390]: Long tail cancellation due to optimal threshold Jan 1 00:09:48.730954500 (632h40m48.602037536s) [noise=true , address=145.79.227.44:4194 , piece_id=VY4CCXV3YXPGQTZC3N2SS5V5ZAT43HAR3CEPU2GSNKZH2TUPRHVA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Jan 1 00:09:48.730970360 (632h40m48.602053395s) [noise=true , address=145.79.227.44:4194 , piece_id=VY4CCXV3YXPGQTZC3N2SS5V5ZAT43HAR3CEPU2GSNKZH2TUPRHVA, seg_pos={0 0}, upload=554390]: Long tail cancellation due to optimal threshold Jan 1 00:09:48.730993675 (632h40m48.60207671s) [seg_pos={0 0}, upload=554390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:09:48.731090710 (632h40m48.602173745s) [upload=554390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:09:48.731109655 (632h40m48.602192721s) [upload=554390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:09:49.025550299 (632h40m48.896633334s) [upload=554390]: Done uploading segments... Jan 1 00:09:49.025568433 (632h40m48.896651468s) [upload=554390]: Upload finished. err: <nil> Duration: 19.25992002s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....................X...........X] duration: 1m0.004573152s trace: https://jaeger.dev.storj.io/trace/67e2279657e45584 FAIL (2026-01-01 00:11:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, Piece ID: Y4TNTQZ2DERVYJ45IWES34IREZM72VWEUJT5DFXBPSFXCXT2ZLBA) context deadline exceeded; piecestore: (Node ID: 12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, Piece ID: 2SZZCPGVL77TPTSZCD7YPVJSFVKJGQHYQAMMGOP23VMEELC6DNSA) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: EJOPAXWICBWSBGS4USQCMHX6U4XWECUEPRJJ3RQIPAMHULYOXOFQ) context deadline exceeded; piecestore: (Node ID: 12ZNCRKd7xMFNHfCJBJ38yfyUmczhYwJetKtiQxX9SDrwxRhzTU, Piece ID: S7XHBNCFCHXSC6XEBXTKXGQV2C3QYOZUNJHXUK32MV62OR7THHYA) context deadline exceeded ======LOGS====== :4936 , piece_id=REO4SYQE655A6OWXGR2IARSVCLQ3OS24DDIQJS5ZSL5VO33ZMGFQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.050418220 (632h41m18.921501255s) [noise=true , address=77.90.31.14:4677 , piece_id=EJOPAXWICBWSBGS4USQCMHX6U4XWECUEPRJJ3RQIPAMHULYOXOFQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.050930676 (632h41m18.922013721s) [noise=true , address=77.90.31.15:4714 , piece_id=Y4TNTQZ2DERVYJ45IWES34IREZM72VWEUJT5DFXBPSFXCXT2ZLBA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.051454794 (632h41m18.922537839s) [noise=true , address=45.91.116.12:4251 , piece_id=S7XHBNCFCHXSC6XEBXTKXGQV2C3QYOZUNJHXUK32MV62OR7THHYA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.051881115 (632h41m18.92296416s) [noise=true , address=64.49.13.140:4509 , piece_id=DR2NS5DIRRU4LHOL7VEIAQZLDM6IPBPCCFULLZSFRJOPVUH36DJQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.052297879 (632h41m18.923380924s) [noise=true , address=64.49.13.140:4511 , piece_id=3HN3335O7DAFTBJA5KQP6UXWZVPZ5VUIB3EFA4NJJWBXRXFJF7JQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.052727466 (632h41m18.923810502s) [noise=true , address=77.90.31.11:4581 , piece_id=NNTFVIMH65B65LCG4B33Y5PW3M2ZTBWM5EPJN7LCXFBIW65C4P3Q, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.053205347 (632h41m18.924288392s) [noise=true , address=145.79.227.28:4760 , piece_id=VELHSBW4LSXCQVAXKZ3XOKF2WNVCL7VUEKB7N6UKKLA5ZPXCVHGA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.054372899 (632h41m18.925455954s) [noise=true , address=77.90.31.18:4824 , piece_id=BRQTN6LIHXSOQNAMIAP5KZC2ZVCK4YV5Y6W2RBSNZLUMAX4I2UWQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.055076449 (632h41m18.926159494s) [noise=true , address=145.79.227.30:4780 , piece_id=K5YFLGQTJVYK5OOU54GMZANYVEFWWVQ3PQ2L4PZUU2WRXR3FPJQQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.055634421 (632h41m18.926717467s) [noise=true , address=64.49.13.26:4954 , piece_id=KBZCJYXHKI2JKZAGP2R52FRRGQLKNEG33ALYLJLALE5JVJEAPGOQ, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.056155594 (632h41m18.927238629s) [noise=true , address=64.49.13.169:4709 , piece_id=SIK5PLNSLWNBRP36CR4GOSDNWBZ42H67TOPQTXSDWXGFMTTBFD5Q, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.056304037 (632h41m18.927387072s) [noise=true , address=145.79.227.24:4588 , piece_id=DGIL7QT2J72ICIIGAQ4NECOPHPBOAHCITAT6SNHGW76X2XKYYC4A, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.056954675 (632h41m18.92803771s) [noise=true , address=77.90.31.12:4619 , piece_id=2SZZCPGVL77TPTSZCD7YPVJSFVKJGQHYQAMMGOP23VMEELC6DNSA, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:19.057515434 (632h41m18.928598479s) [noise=true , address=145.79.227.14:4864 , piece_id=G4UO6U435XHRMCXQSMAIUVK5SODNPU7FBJZG4ZNJDFVGQWCHPZ3A, seg_pos={0 0}, upload=554397]: Uploading piece... Jan 1 00:10:20.496658364 (632h41m20.367741399s) [upload=554397]: Next returned nil segment Jan 1 00:10:20.496673834 (632h41m20.367756869s) [upload=554397]: Waiting for error group managing segments... Jan 1 00:10:20.806767677 (632h41m20.677850712s) [noise=true , address=64.49.13.140:4511 , piece_id=3HN3335O7DAFTBJA5KQP6UXWZVPZ5VUIB3EFA4NJJWBXRXFJF7JQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:20.947121307 (632h41m20.818204352s) [noise=true , address=145.79.227.28:4760 , piece_id=VELHSBW4LSXCQVAXKZ3XOKF2WNVCL7VUEKB7N6UKKLA5ZPXCVHGA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:20.977079480 (632h41m20.848162525s) [noise=true , address=64.49.13.140:4509 , piece_id=DR2NS5DIRRU4LHOL7VEIAQZLDM6IPBPCCFULLZSFRJOPVUH36DJQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.101757722 (632h41m20.972840757s) [noise=true , address=64.49.13.26:4954 , piece_id=KBZCJYXHKI2JKZAGP2R52FRRGQLKNEG33ALYLJLALE5JVJEAPGOQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.124455562 (632h41m20.995538607s) [noise=true , address=77.90.31.12:4619 , piece_id=2SZZCPGVL77TPTSZCD7YPVJSFVKJGQHYQAMMGOP23VMEELC6DNSA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.166319495 (632h41m21.03740253s) [noise=true , address=77.90.31.11:4581 , piece_id=NNTFVIMH65B65LCG4B33Y5PW3M2ZTBWM5EPJN7LCXFBIW65C4P3Q, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.214239851 (632h41m21.085322886s) [noise=true , address=77.90.31.14:4677 , piece_id=EJOPAXWICBWSBGS4USQCMHX6U4XWECUEPRJJ3RQIPAMHULYOXOFQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.231422007 (632h41m21.102505052s) [noise=true , address=77.90.31.15:4714 , piece_id=Y4TNTQZ2DERVYJ45IWES34IREZM72VWEUJT5DFXBPSFXCXT2ZLBA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.340538969 (632h41m21.211622004s) [noise=true , address=45.91.116.12:4251 , piece_id=S7XHBNCFCHXSC6XEBXTKXGQV2C3QYOZUNJHXUK32MV62OR7THHYA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.642906661 (632h41m21.513989706s) [noise=true , address=145.79.227.14:4864 , piece_id=G4UO6U435XHRMCXQSMAIUVK5SODNPU7FBJZG4ZNJDFVGQWCHPZ3A, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.820646487 (632h41m21.691729532s) [noise=true , address=145.79.227.44:4196 , piece_id=K4EPTOZT3VLMXSZYESGAFHBLVIWA2ULE2VLU37SCJWP7ZMI4LIRQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.856562155 (632h41m21.7276452s) [noise=true , address=145.79.227.30:4780 , piece_id=K5YFLGQTJVYK5OOU54GMZANYVEFWWVQ3PQ2L4PZUU2WRXR3FPJQQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:21.919986602 (632h41m21.791069647s) [noise=true , address=145.79.227.34:4936 , piece_id=REO4SYQE655A6OWXGR2IARSVCLQ3OS24DDIQJS5ZSL5VO33ZMGFQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:22.039935498 (632h41m21.911018543s) [noise=true , address=77.90.31.18:4824 , piece_id=BRQTN6LIHXSOQNAMIAP5KZC2ZVCK4YV5Y6W2RBSNZLUMAX4I2UWQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: <nil> Jan 1 00:10:22.039959233 (632h41m21.911042288s) [seg_pos={0 0}, upload=554397]: Segment reached optimal threshold of 14 pieces. Jan 1 00:10:22.040191375 (632h41m21.91127441s) [noise=true , address=145.79.227.33:4877 , piece_id=Z4OYPOCWRMF6L3MDB4TO6I5LPW33SLGZVMLVGFQQOZE2OAE4EYIQ, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Jan 1 00:10:22.040202396 (632h41m21.911285441s) [noise=true , address=145.79.227.33:4877 , piece_id=Z4OYPOCWRMF6L3MDB4TO6I5LPW33SLGZVMLVGFQQOZE2OAE4EYIQ, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040277219 (632h41m21.911360274s) [noise=true , address=145.79.227.5:4243 , piece_id=RSEPWUQ5TT3NEXUT2YYTNZN5BJQQ24ED2H7DGMV2MMBB3E3TKWCA, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FAn2HGJpXP5dfbYfL8PkxCfrBM3SHJVCASQMSy1kaRqLhyiRv): protocol: write tcp 188.166.225.170:53800->145.79.227.5:4243: use of closed network connection; write tcp 188.166.225.170:53800->145.79.227.5:4243: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53800->145.79.227.5:4243: use of closed network connection Jan 1 00:10:22.040288921 (632h41m21.911372176s) [noise=true , address=145.79.227.5:4243 , piece_id=RSEPWUQ5TT3NEXUT2YYTNZN5BJQQ24ED2H7DGMV2MMBB3E3TKWCA, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040324859 (632h41m21.911407894s) [noise=true , address=64.49.13.169:4709 , piece_id=SIK5PLNSLWNBRP36CR4GOSDNWBZ42H67TOPQTXSDWXGFMTTBFD5Q, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:42120->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:42120->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42120->64.49.13.169:4709: use of closed network connection Jan 1 00:10:22.040332203 (632h41m21.911415248s) [noise=true , address=64.49.13.169:4709 , piece_id=SIK5PLNSLWNBRP36CR4GOSDNWBZ42H67TOPQTXSDWXGFMTTBFD5Q, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040454306 (632h41m21.911537341s) [noise=true , address=145.79.227.24:4588 , piece_id=DGIL7QT2J72ICIIGAQ4NECOPHPBOAHCITAT6SNHGW76X2XKYYC4A, seg_pos={0 0}, upload=554397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Jan 1 00:10:22.040461780 (632h41m21.911544815s) [noise=true , address=145.79.227.24:4588 , piece_id=DGIL7QT2J72ICIIGAQ4NECOPHPBOAHCITAT6SNHGW76X2XKYYC4A, seg_pos={0 0}, upload=554397]: Long tail cancellation due to optimal threshold Jan 1 00:10:22.040557783 (632h41m21.911640828s) [seg_pos={0 0}, upload=554397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:10:22.040638456 (632h41m21.911721491s) [upload=554397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:10:22.040644648 (632h41m21.911727693s) [upload=554397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:10:22.331369772 (632h41m22.202452807s) [upload=554397]: Done uploading segments... Jan 1 00:10:22.331387687 (632h41m22.202470732s) [upload=554397]: Upload finished. err: <nil> Duration: 1m0.004599072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X....X...X............X....X....X......X....X....X.....XXX.] duration: 18.989064557s trace: https://jaeger.dev.storj.io/trace/23758042101091ca FAIL (2026-01-01 00:10:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 00:09:45.465120922 (632h40m45.336203977s) [upload=554390]: Starting upload Jan 1 00:09:45.465208038 (632h40m45.336291083s) [upload=554390]: Done starting upload Jan 1 00:09:45.465279364 (632h40m45.336362409s) [upload=554390]: Uploading segments... Jan 1 00:09:45.465460078 (632h40m45.336543123s) [upload=554390]: Got next segment. Inline: false Jan 1 00:09:45.465539890 (632h40m45.336622935s) [upload=554390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:09:45.465550621 (632h40m45.336633666s) [upload=554390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:09:45.715683168 (632h40m45.586766203s) [seg_pos={0 0}, upload=554390]: Begin upload segment... Jan 1 00:09:45.715733694 (632h40m45.586816729s) [seg_pos={0 0}, upload=554390]: Done begin upload segment. Jan 1 00:09:45.715765865 (632h40m45.5868489s) [noise=true , address=145.79.227.44:4194 , piece_id=VY4CCXV3YXPGQTZC3N2SS5V5ZAT43HAR3CEPU2GSNKZH2TUPRHVA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.715781495 (632h40m45.58686453s) [noise=true , address=145.79.227.36:4519 , piece_id=CBBMCLRSIGUWSXADVOV2EQEC7RCIZN7JSQTJAFJADMWVXWEVJ2JA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.716265758 (632h40m45.587348793s) [noise=true , address=145.79.227.42:4123 , piece_id=SHPQ3LZ46F5MLCAP6574MXPXCWOYHVD2CGWW7OXYMWQ2D2TCEJ4Q, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.716650530 (632h40m45.587733565s) [noise=true , address=145.79.227.4:4187 , piece_id=TFHN6TIXVMM2OWBS3IA3624UEUB6WMNRQILG3JVBTDCMX5XH4VEA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717004985 (632h40m45.58808802s) [noise=true , address=45.91.116.7:4012 , piece_id=EVWNGG3YM7I23OIRI46V7VUGTV45G2EVBLJS3QBBB3YJYPIG3W4A, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717333260 (632h40m45.588416305s) [noise=true , address=145.79.227.26:4655 , piece_id=IZSEUCFPZZZTUA53C27DAE7DUTYUZDOFD3QYKYZWXH6IOW7NBB6Q, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717573066 (632h40m45.588656111s) [noise=true , address=77.90.31.17:4770 , piece_id=NRFQJP7ZYK6OY73AVLRJT3BSLQNHZJFQPJEID46D6FJ47LUH4AEA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717926149 (632h40m45.589009184s) [noise=true , address=45.91.116.13:4154 , piece_id=U4HM6NKCSACXDKIVCFNDBBWSPAR3XBXSQXGUC7S7KMOZ4K3F6WAQ, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718271356 (632h40m45.589354391s) [noise=true , address=145.79.227.11:4730 , piece_id=CMMVQOCGO7BLPGOL5MGZTTLICEOOPVRF4OYDRYQAPBIJPC5GHVXQ, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718453132 (632h40m45.589536167s) [noise=true , address=77.90.31.12:4590 , piece_id=DXF6AZXWRRGFCR7L2O4BB67W2IIK6AYOZGL4YDXH27EQF35GIAGA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718639016 (632h40m45.589722052s) [noise=true , address=145.79.227.6:4259 , piece_id=GNSHMUVXCYOOM4JVTPDJJA4VUZJDQPPMDBQW6F5R62WL3WHVWPDQ, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718954607 (632h40m45.590037652s) [noise=true , address=77.90.31.15:4700 , piece_id=XUBRSQ6NXMJ44NQOGXSRIFAWRLPE73YURXV5LMVZAZPL3EQPQSPA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718960418 (632h40m45.590043463s) [noise=true , address=77.90.31.21:4880 , piece_id=MTSDTM75BWGDVUU2BNR6N3SR5S4TJALRQU6PTNW4XGPSI6Q24FIA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.719254398 (632h40m45.590337433s) [noise=true , address=145.79.227.17:4972 , piece_id=FBZO5BZ3LOTOBL7WJUI4KILPWZIF4WISNDK4HLZX6PNAOSOHN6PA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.719346163 (632h40m45.590429198s) [noise=true , address=45.91.116.4:4937 , piece_id=S76Z4ZQLXAZCBAYSUB6JNZXHJ3HSZJEIHWRR24HDN34QDM6WGIWA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.719546875 (632h40m45.59062992s) [noise=true , address=45.91.116.10:4084 , piece_id=J6CCFXCKYZGJBGTX6H7VGC3JYPG5HUQ5JX666VTVDY4NJ3BG444Q, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:47.414914601 (632h40m47.285997636s) [upload=554390]: Next returned nil segment Jan 1 00:09:47.414928316 (632h40m47.286011352s) [upload=554390]: Waiting for error group managing segments... Jan 1 00:09:47.696668918 (632h40m47.567751953s) [noise=true , address=145.79.227.4:4187 , piece_id=TFHN6TIXVMM2OWBS3IA3624UEUB6WMNRQILG3JVBTDCMX5XH4VEA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.700534478 (632h40m47.571617523s) [noise=true , address=145.79.227.11:4730 , piece_id=CMMVQOCGO7BLPGOL5MGZTTLICEOOPVRF4OYDRYQAPBIJPC5GHVXQ, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.703399113 (632h40m47.574482148s) [noise=true , address=145.79.227.17:4972 , piece_id=FBZO5BZ3LOTOBL7WJUI4KILPWZIF4WISNDK4HLZX6PNAOSOHN6PA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.712386226 (632h40m47.583469271s) [noise=true , address=145.79.227.42:4123 , piece_id=SHPQ3LZ46F5MLCAP6574MXPXCWOYHVD2CGWW7OXYMWQ2D2TCEJ4Q, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.720236954 (632h40m47.591319989s) [noise=true , address=145.79.227.36:4519 , piece_id=CBBMCLRSIGUWSXADVOV2EQEC7RCIZN7JSQTJAFJADMWVXWEVJ2JA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.735226475 (632h40m47.60630951s) [noise=true , address=145.79.227.6:4259 , piece_id=GNSHMUVXCYOOM4JVTPDJJA4VUZJDQPPMDBQW6F5R62WL3WHVWPDQ, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.737387791 (632h40m47.608470826s) [noise=true , address=145.79.227.26:4655 , piece_id=IZSEUCFPZZZTUA53C27DAE7DUTYUZDOFD3QYKYZWXH6IOW7NBB6Q, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.885902698 (632h40m47.756985734s) [noise=true , address=77.90.31.12:4590 , piece_id=DXF6AZXWRRGFCR7L2O4BB67W2IIK6AYOZGL4YDXH27EQF35GIAGA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.994421789 (632h40m47.865504824s) [noise=true , address=45.91.116.13:4154 , piece_id=U4HM6NKCSACXDKIVCFNDBBWSPAR3XBXSQXGUC7S7KMOZ4K3F6WAQ, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.004618224 (632h40m47.875701269s) [noise=true , address=77.90.31.15:4700 , piece_id=XUBRSQ6NXMJ44NQOGXSRIFAWRLPE73YURXV5LMVZAZPL3EQPQSPA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.071212837 (632h40m47.942295882s) [noise=true , address=45.91.116.4:4937 , piece_id=S76Z4ZQLXAZCBAYSUB6JNZXHJ3HSZJEIHWRR24HDN34QDM6WGIWA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.077205898 (632h40m47.948288933s) [noise=true , address=45.91.116.10:4084 , piece_id=J6CCFXCKYZGJBGTX6H7VGC3JYPG5HUQ5JX666VTVDY4NJ3BG444Q, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.730471781 (632h40m48.601554816s) [noise=true , address=45.91.116.7:4012 , piece_id=EVWNGG3YM7I23OIRI46V7VUGTV45G2EVBLJS3QBBB3YJYPIG3W4A, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.730493042 (632h40m48.601576077s) [seg_pos={0 0}, upload=554390]: Segment reached optimal threshold of 13 pieces. Jan 1 00:09:48.730733249 (632h40m48.601816294s) [noise=true , address=77.90.31.21:4880 , piece_id=MTSDTM75BWGDVUU2BNR6N3SR5S4TJALRQU6PTNW4XGPSI6Q24FIA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Jan 1 00:09:48.730739471 (632h40m48.601822506s) [noise=true , address=77.90.31.21:4880 , piece_id=MTSDTM75BWGDVUU2BNR6N3SR5S4TJALRQU6PTNW4XGPSI6Q24FIA, seg_pos={0 0}, upload=554390]: Long tail cancellation due to optimal threshold Jan 1 00:09:48.730936556 (632h40m48.602019601s) [noise=true , address=77.90.31.17:4770 , piece_id=NRFQJP7ZYK6OY73AVLRJT3BSLQNHZJFQPJEID46D6FJ47LUH4AEA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 1 00:09:48.730943088 (632h40m48.602026133s) [noise=true , address=77.90.31.17:4770 , piece_id=NRFQJP7ZYK6OY73AVLRJT3BSLQNHZJFQPJEID46D6FJ47LUH4AEA, seg_pos={0 0}, upload=554390]: Long tail cancellation due to optimal threshold Jan 1 00:09:48.730954500 (632h40m48.602037536s) [noise=true , address=145.79.227.44:4194 , piece_id=VY4CCXV3YXPGQTZC3N2SS5V5ZAT43HAR3CEPU2GSNKZH2TUPRHVA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Jan 1 00:09:48.730970360 (632h40m48.602053395s) [noise=true , address=145.79.227.44:4194 , piece_id=VY4CCXV3YXPGQTZC3N2SS5V5ZAT43HAR3CEPU2GSNKZH2TUPRHVA, seg_pos={0 0}, upload=554390]: Long tail cancellation due to optimal threshold Jan 1 00:09:48.730993675 (632h40m48.60207671s) [seg_pos={0 0}, upload=554390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:09:48.731090710 (632h40m48.602173745s) [upload=554390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:09:48.731109655 (632h40m48.602192721s) [upload=554390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:09:49.025550299 (632h40m48.896633334s) [upload=554390]: Done uploading segments... Jan 1 00:09:49.025568433 (632h40m48.896651468s) [upload=554390]: Upload finished. err: <nil> Duration: 18.989084144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X....X...X............X....X....X......X....X....X.....XXX] duration: 30.004249091s trace: https://jaeger.dev.storj.io/trace/23758042101091ca FAIL (2026-01-01 00:10:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 00:09:45.465120922 (632h40m45.336203977s) [upload=554390]: Starting upload Jan 1 00:09:45.465208038 (632h40m45.336291083s) [upload=554390]: Done starting upload Jan 1 00:09:45.465279364 (632h40m45.336362409s) [upload=554390]: Uploading segments... Jan 1 00:09:45.465460078 (632h40m45.336543123s) [upload=554390]: Got next segment. Inline: false Jan 1 00:09:45.465539890 (632h40m45.336622935s) [upload=554390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:09:45.465550621 (632h40m45.336633666s) [upload=554390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:09:45.715683168 (632h40m45.586766203s) [seg_pos={0 0}, upload=554390]: Begin upload segment... Jan 1 00:09:45.715733694 (632h40m45.586816729s) [seg_pos={0 0}, upload=554390]: Done begin upload segment. Jan 1 00:09:45.715765865 (632h40m45.5868489s) [noise=true , address=145.79.227.44:4194 , piece_id=VY4CCXV3YXPGQTZC3N2SS5V5ZAT43HAR3CEPU2GSNKZH2TUPRHVA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.715781495 (632h40m45.58686453s) [noise=true , address=145.79.227.36:4519 , piece_id=CBBMCLRSIGUWSXADVOV2EQEC7RCIZN7JSQTJAFJADMWVXWEVJ2JA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.716265758 (632h40m45.587348793s) [noise=true , address=145.79.227.42:4123 , piece_id=SHPQ3LZ46F5MLCAP6574MXPXCWOYHVD2CGWW7OXYMWQ2D2TCEJ4Q, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.716650530 (632h40m45.587733565s) [noise=true , address=145.79.227.4:4187 , piece_id=TFHN6TIXVMM2OWBS3IA3624UEUB6WMNRQILG3JVBTDCMX5XH4VEA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717004985 (632h40m45.58808802s) [noise=true , address=45.91.116.7:4012 , piece_id=EVWNGG3YM7I23OIRI46V7VUGTV45G2EVBLJS3QBBB3YJYPIG3W4A, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717333260 (632h40m45.588416305s) [noise=true , address=145.79.227.26:4655 , piece_id=IZSEUCFPZZZTUA53C27DAE7DUTYUZDOFD3QYKYZWXH6IOW7NBB6Q, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717573066 (632h40m45.588656111s) [noise=true , address=77.90.31.17:4770 , piece_id=NRFQJP7ZYK6OY73AVLRJT3BSLQNHZJFQPJEID46D6FJ47LUH4AEA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.717926149 (632h40m45.589009184s) [noise=true , address=45.91.116.13:4154 , piece_id=U4HM6NKCSACXDKIVCFNDBBWSPAR3XBXSQXGUC7S7KMOZ4K3F6WAQ, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718271356 (632h40m45.589354391s) [noise=true , address=145.79.227.11:4730 , piece_id=CMMVQOCGO7BLPGOL5MGZTTLICEOOPVRF4OYDRYQAPBIJPC5GHVXQ, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718453132 (632h40m45.589536167s) [noise=true , address=77.90.31.12:4590 , piece_id=DXF6AZXWRRGFCR7L2O4BB67W2IIK6AYOZGL4YDXH27EQF35GIAGA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718639016 (632h40m45.589722052s) [noise=true , address=145.79.227.6:4259 , piece_id=GNSHMUVXCYOOM4JVTPDJJA4VUZJDQPPMDBQW6F5R62WL3WHVWPDQ, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718954607 (632h40m45.590037652s) [noise=true , address=77.90.31.15:4700 , piece_id=XUBRSQ6NXMJ44NQOGXSRIFAWRLPE73YURXV5LMVZAZPL3EQPQSPA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.718960418 (632h40m45.590043463s) [noise=true , address=77.90.31.21:4880 , piece_id=MTSDTM75BWGDVUU2BNR6N3SR5S4TJALRQU6PTNW4XGPSI6Q24FIA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.719254398 (632h40m45.590337433s) [noise=true , address=145.79.227.17:4972 , piece_id=FBZO5BZ3LOTOBL7WJUI4KILPWZIF4WISNDK4HLZX6PNAOSOHN6PA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.719346163 (632h40m45.590429198s) [noise=true , address=45.91.116.4:4937 , piece_id=S76Z4ZQLXAZCBAYSUB6JNZXHJ3HSZJEIHWRR24HDN34QDM6WGIWA, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:45.719546875 (632h40m45.59062992s) [noise=true , address=45.91.116.10:4084 , piece_id=J6CCFXCKYZGJBGTX6H7VGC3JYPG5HUQ5JX666VTVDY4NJ3BG444Q, seg_pos={0 0}, upload=554390]: Uploading piece... Jan 1 00:09:47.414914601 (632h40m47.285997636s) [upload=554390]: Next returned nil segment Jan 1 00:09:47.414928316 (632h40m47.286011352s) [upload=554390]: Waiting for error group managing segments... Jan 1 00:09:47.696668918 (632h40m47.567751953s) [noise=true , address=145.79.227.4:4187 , piece_id=TFHN6TIXVMM2OWBS3IA3624UEUB6WMNRQILG3JVBTDCMX5XH4VEA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.700534478 (632h40m47.571617523s) [noise=true , address=145.79.227.11:4730 , piece_id=CMMVQOCGO7BLPGOL5MGZTTLICEOOPVRF4OYDRYQAPBIJPC5GHVXQ, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.703399113 (632h40m47.574482148s) [noise=true , address=145.79.227.17:4972 , piece_id=FBZO5BZ3LOTOBL7WJUI4KILPWZIF4WISNDK4HLZX6PNAOSOHN6PA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.712386226 (632h40m47.583469271s) [noise=true , address=145.79.227.42:4123 , piece_id=SHPQ3LZ46F5MLCAP6574MXPXCWOYHVD2CGWW7OXYMWQ2D2TCEJ4Q, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.720236954 (632h40m47.591319989s) [noise=true , address=145.79.227.36:4519 , piece_id=CBBMCLRSIGUWSXADVOV2EQEC7RCIZN7JSQTJAFJADMWVXWEVJ2JA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.735226475 (632h40m47.60630951s) [noise=true , address=145.79.227.6:4259 , piece_id=GNSHMUVXCYOOM4JVTPDJJA4VUZJDQPPMDBQW6F5R62WL3WHVWPDQ, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.737387791 (632h40m47.608470826s) [noise=true , address=145.79.227.26:4655 , piece_id=IZSEUCFPZZZTUA53C27DAE7DUTYUZDOFD3QYKYZWXH6IOW7NBB6Q, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.885902698 (632h40m47.756985734s) [noise=true , address=77.90.31.12:4590 , piece_id=DXF6AZXWRRGFCR7L2O4BB67W2IIK6AYOZGL4YDXH27EQF35GIAGA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:47.994421789 (632h40m47.865504824s) [noise=true , address=45.91.116.13:4154 , piece_id=U4HM6NKCSACXDKIVCFNDBBWSPAR3XBXSQXGUC7S7KMOZ4K3F6WAQ, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.004618224 (632h40m47.875701269s) [noise=true , address=77.90.31.15:4700 , piece_id=XUBRSQ6NXMJ44NQOGXSRIFAWRLPE73YURXV5LMVZAZPL3EQPQSPA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.071212837 (632h40m47.942295882s) [noise=true , address=45.91.116.4:4937 , piece_id=S76Z4ZQLXAZCBAYSUB6JNZXHJ3HSZJEIHWRR24HDN34QDM6WGIWA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.077205898 (632h40m47.948288933s) [noise=true , address=45.91.116.10:4084 , piece_id=J6CCFXCKYZGJBGTX6H7VGC3JYPG5HUQ5JX666VTVDY4NJ3BG444Q, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.730471781 (632h40m48.601554816s) [noise=true , address=45.91.116.7:4012 , piece_id=EVWNGG3YM7I23OIRI46V7VUGTV45G2EVBLJS3QBBB3YJYPIG3W4A, seg_pos={0 0}, upload=554390]: Done uploading piece. err: <nil> Jan 1 00:09:48.730493042 (632h40m48.601576077s) [seg_pos={0 0}, upload=554390]: Segment reached optimal threshold of 13 pieces. Jan 1 00:09:48.730733249 (632h40m48.601816294s) [noise=true , address=77.90.31.21:4880 , piece_id=MTSDTM75BWGDVUU2BNR6N3SR5S4TJALRQU6PTNW4XGPSI6Q24FIA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Jan 1 00:09:48.730739471 (632h40m48.601822506s) [noise=true , address=77.90.31.21:4880 , piece_id=MTSDTM75BWGDVUU2BNR6N3SR5S4TJALRQU6PTNW4XGPSI6Q24FIA, seg_pos={0 0}, upload=554390]: Long tail cancellation due to optimal threshold Jan 1 00:09:48.730936556 (632h40m48.602019601s) [noise=true , address=77.90.31.17:4770 , piece_id=NRFQJP7ZYK6OY73AVLRJT3BSLQNHZJFQPJEID46D6FJ47LUH4AEA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 1 00:09:48.730943088 (632h40m48.602026133s) [noise=true , address=77.90.31.17:4770 , piece_id=NRFQJP7ZYK6OY73AVLRJT3BSLQNHZJFQPJEID46D6FJ47LUH4AEA, seg_pos={0 0}, upload=554390]: Long tail cancellation due to optimal threshold Jan 1 00:09:48.730954500 (632h40m48.602037536s) [noise=true , address=145.79.227.44:4194 , piece_id=VY4CCXV3YXPGQTZC3N2SS5V5ZAT43HAR3CEPU2GSNKZH2TUPRHVA, seg_pos={0 0}, upload=554390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Jan 1 00:09:48.730970360 (632h40m48.602053395s) [noise=true , address=145.79.227.44:4194 , piece_id=VY4CCXV3YXPGQTZC3N2SS5V5ZAT43HAR3CEPU2GSNKZH2TUPRHVA, seg_pos={0 0}, upload=554390]: Long tail cancellation due to optimal threshold Jan 1 00:09:48.730993675 (632h40m48.60207671s) [seg_pos={0 0}, upload=554390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:09:48.731090710 (632h40m48.602173745s) [upload=554390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:09:48.731109655 (632h40m48.602192721s) [upload=554390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:09:49.025550299 (632h40m48.896633334s) [upload=554390]: Done uploading segments... Jan 1 00:09:49.025568433 (632h40m48.896651468s) [upload=554390]: Upload finished. err: <nil> Duration: 30.004270192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........................X....................X...........] duration: 18.619300464s trace: https://jaeger.dev.storj.io/trace/71cc346ea8b1cbd1 FAIL (2026-01-01 00:10:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, Piece ID: YQSVWEYT4QAJPRK5FQUUF3U7IHUD2OTTMERFDD3XSJAM44XBKU6Q) context deadline exceeded; piecestore: (Node ID: 1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, Piece ID: NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ) context deadline exceeded; piecestore: (Node ID: 12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof, Piece ID: OAJRVQHDC5T7C25HBIWZIWZXBFXPRHLAKUOC7RZUF6HI5N4PJINQ) context deadline exceeded ======LOGS====== Z4BAFLDDP62G2AEZ4QUDUUFYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.221779207 (632h31m26.092862242s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.221794527 (632h31m26.092877572s) [noise=true , address=145.79.227.43:4168 , piece_id=67E2PXGLNBVDVAWNMTIJ5NPIZ5KUFOSKOJVGMP7V5ULZQ2QO2HNA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222236768 (632h31m26.093319814s) [noise=true , address=145.79.227.19:4104 , piece_id=TRLT4BXD66MUVFYX7H35XP2PJJ3KN5F2HPHPN66L7KVK42ANCQYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222256006 (632h31m26.093339051s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222688048 (632h31m26.093771093s) [noise=true , address=77.90.31.13:4634 , piece_id=NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.223053204 (632h31m26.094136249s) [noise=true , address=145.79.227.49:4372 , piece_id=3DNK6HF64TE4MU6M7AWTNW2ILTXNGNMPAX5HLKMROL77R5YJZIYA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.223504623 (632h31m26.094587668s) [noise=true , address=77.90.31.14:4673 , piece_id=FEZGA5U275AGSYP6L2DBFWTGCHC5S6Y4APJ7WGZ6NNTPP4E3QRBQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224091681 (632h31m26.095174726s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224459571 (632h31m26.095542616s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224880994 (632h31m26.095964039s) [noise=true , address=45.91.116.13:4164 , piece_id=5NAIJXVZRI2ZK256AELB7M6Z4PZRELIL6H35EVOQXJK7444QWXAQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.225276838 (632h31m26.096359883s) [noise=true , address=193.149.164.143:4698 , piece_id=JLTXGS4WB7RT7KA5DCX5XFFUZNXOMSR7EBBQJRLGZRJWKVRWQBOA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.226050822 (632h31m26.097133897s) [noise=true , address=145.79.227.34:4929 , piece_id=A7XSTYBGFCLEBSL7GU3N4RGPQB6MPEYJLZQQM7RIG4MEDRG5RJWQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.228223077 (632h31m26.099306123s) [noise=true , address=145.79.227.33:4879 , piece_id=G3PEREJSNH3YF4KS2HYQO7GUFGTAA5ZKC2K5MVHPQO2EZNXBQSCQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:27.694323976 (632h31m27.565407021s) [upload=554251]: Next returned nil segment Jan 1 00:00:27.694348883 (632h31m27.565431928s) [upload=554251]: Waiting for error group managing segments... Jan 1 00:00:28.086794748 (632h31m27.957877793s) [noise=true , address=145.79.227.43:4168 , piece_id=67E2PXGLNBVDVAWNMTIJ5NPIZ5KUFOSKOJVGMP7V5ULZQ2QO2HNA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.105443428 (632h31m27.976526473s) [noise=true , address=145.79.227.19:4104 , piece_id=TRLT4BXD66MUVFYX7H35XP2PJJ3KN5F2HPHPN66L7KVK42ANCQYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.229480186 (632h31m28.100563231s) [noise=true , address=77.90.31.10:4529 , piece_id=SFBOVPQEXSGZ2H5YSXKIQO65NRCVIBOW3T6EAW6NUOXDS3A2X2VA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.273850481 (632h31m28.144933526s) [noise=true , address=64.49.13.154:4926 , piece_id=QGUG2VZT56RC22UM2ZGPLELFJTFZ4BAFLDDP62G2AEZ4QUDUUFYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.291760373 (632h31m28.162843448s) [noise=true , address=77.90.31.13:4634 , piece_id=NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.353007052 (632h31m28.224090097s) [noise=true , address=77.90.31.14:4673 , piece_id=FEZGA5U275AGSYP6L2DBFWTGCHC5S6Y4APJ7WGZ6NNTPP4E3QRBQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.386749041 (632h31m28.257832086s) [noise=true , address=45.91.116.13:4164 , piece_id=5NAIJXVZRI2ZK256AELB7M6Z4PZRELIL6H35EVOQXJK7444QWXAQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.459684066 (632h31m28.330767111s) [noise=true , address=145.79.227.33:4879 , piece_id=G3PEREJSNH3YF4KS2HYQO7GUFGTAA5ZKC2K5MVHPQO2EZNXBQSCQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.754224531 (632h31m28.625307576s) [noise=true , address=45.91.116.5:4319 , piece_id=OAJRVQHDC5T7C25HBIWZIWZXBFXPRHLAKUOC7RZUF6HI5N4PJINQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.012527725 (632h31m28.88361077s) [noise=true , address=145.79.227.30:4792 , piece_id=3PSRSL6JHSXPJYLM6CHKL4XDSROFLBVWTFYPL42SIPJQSPM6AJFQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.044467761 (632h31m28.915550806s) [noise=true , address=145.79.227.49:4372 , piece_id=3DNK6HF64TE4MU6M7AWTNW2ILTXNGNMPAX5HLKMROL77R5YJZIYA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.107424457 (632h31m28.978507502s) [noise=true , address=45.91.116.14:4197 , piece_id=YQSVWEYT4QAJPRK5FQUUF3U7IHUD2OTTMERFDD3XSJAM44XBKU6Q, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.132068652 (632h31m29.003151697s) [noise=true , address=145.79.227.34:4929 , piece_id=A7XSTYBGFCLEBSL7GU3N4RGPQB6MPEYJLZQQM7RIG4MEDRG5RJWQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.337855852 (632h31m29.208938887s) [noise=true , address=193.149.164.143:4698 , piece_id=JLTXGS4WB7RT7KA5DCX5XFFUZNXOMSR7EBBQJRLGZRJWKVRWQBOA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.337884647 (632h31m29.208967702s) [seg_pos={0 0}, upload=554251]: Segment reached optimal threshold of 14 pieces. Jan 1 00:00:29.338100017 (632h31m29.209183062s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF): protocol: write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection; write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection Jan 1 00:00:29.338108012 (632h31m29.209191057s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338168928 (632h31m29.209251993s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jan 1 00:00:29.338180740 (632h31m29.209263795s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338254290 (632h31m29.209337345s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jvSGi6s5yi55BBFJ4aErrap64wGP2WbDL9oUB6pdGiv8DM385): protocol: write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection; write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection Jan 1 00:00:29.338262516 (632h31m29.209345561s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338380130 (632h31m29.209463175s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection Jan 1 00:00:29.338387354 (632h31m29.209470399s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338772687 (632h31m29.209855732s) [seg_pos={0 0}, upload=554251]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:00:29.338880973 (632h31m29.209964019s) [upload=554251]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:00:29.338887636 (632h31m29.209970681s) [upload=554251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:00:29.640167090 (632h31m29.511250145s) [upload=554251]: Done uploading segments... Jan 1 00:00:29.640190766 (632h31m29.511273811s) [upload=554251]: Upload finished. err: <nil> Duration: 18.619327997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..........................X....................X..........] duration: 31.964422625s trace: https://jaeger.dev.storj.io/trace/71cc346ea8b1cbd1 FAIL (2026-01-01 00:09:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, Piece ID: YQSVWEYT4QAJPRK5FQUUF3U7IHUD2OTTMERFDD3XSJAM44XBKU6Q) context deadline exceeded; piecestore: (Node ID: 1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, Piece ID: NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ) context deadline exceeded; piecestore: (Node ID: 12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof, Piece ID: OAJRVQHDC5T7C25HBIWZIWZXBFXPRHLAKUOC7RZUF6HI5N4PJINQ) context deadline exceeded ======LOGS====== Z4BAFLDDP62G2AEZ4QUDUUFYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.221779207 (632h31m26.092862242s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.221794527 (632h31m26.092877572s) [noise=true , address=145.79.227.43:4168 , piece_id=67E2PXGLNBVDVAWNMTIJ5NPIZ5KUFOSKOJVGMP7V5ULZQ2QO2HNA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222236768 (632h31m26.093319814s) [noise=true , address=145.79.227.19:4104 , piece_id=TRLT4BXD66MUVFYX7H35XP2PJJ3KN5F2HPHPN66L7KVK42ANCQYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222256006 (632h31m26.093339051s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222688048 (632h31m26.093771093s) [noise=true , address=77.90.31.13:4634 , piece_id=NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.223053204 (632h31m26.094136249s) [noise=true , address=145.79.227.49:4372 , piece_id=3DNK6HF64TE4MU6M7AWTNW2ILTXNGNMPAX5HLKMROL77R5YJZIYA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.223504623 (632h31m26.094587668s) [noise=true , address=77.90.31.14:4673 , piece_id=FEZGA5U275AGSYP6L2DBFWTGCHC5S6Y4APJ7WGZ6NNTPP4E3QRBQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224091681 (632h31m26.095174726s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224459571 (632h31m26.095542616s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224880994 (632h31m26.095964039s) [noise=true , address=45.91.116.13:4164 , piece_id=5NAIJXVZRI2ZK256AELB7M6Z4PZRELIL6H35EVOQXJK7444QWXAQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.225276838 (632h31m26.096359883s) [noise=true , address=193.149.164.143:4698 , piece_id=JLTXGS4WB7RT7KA5DCX5XFFUZNXOMSR7EBBQJRLGZRJWKVRWQBOA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.226050822 (632h31m26.097133897s) [noise=true , address=145.79.227.34:4929 , piece_id=A7XSTYBGFCLEBSL7GU3N4RGPQB6MPEYJLZQQM7RIG4MEDRG5RJWQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.228223077 (632h31m26.099306123s) [noise=true , address=145.79.227.33:4879 , piece_id=G3PEREJSNH3YF4KS2HYQO7GUFGTAA5ZKC2K5MVHPQO2EZNXBQSCQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:27.694323976 (632h31m27.565407021s) [upload=554251]: Next returned nil segment Jan 1 00:00:27.694348883 (632h31m27.565431928s) [upload=554251]: Waiting for error group managing segments... Jan 1 00:00:28.086794748 (632h31m27.957877793s) [noise=true , address=145.79.227.43:4168 , piece_id=67E2PXGLNBVDVAWNMTIJ5NPIZ5KUFOSKOJVGMP7V5ULZQ2QO2HNA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.105443428 (632h31m27.976526473s) [noise=true , address=145.79.227.19:4104 , piece_id=TRLT4BXD66MUVFYX7H35XP2PJJ3KN5F2HPHPN66L7KVK42ANCQYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.229480186 (632h31m28.100563231s) [noise=true , address=77.90.31.10:4529 , piece_id=SFBOVPQEXSGZ2H5YSXKIQO65NRCVIBOW3T6EAW6NUOXDS3A2X2VA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.273850481 (632h31m28.144933526s) [noise=true , address=64.49.13.154:4926 , piece_id=QGUG2VZT56RC22UM2ZGPLELFJTFZ4BAFLDDP62G2AEZ4QUDUUFYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.291760373 (632h31m28.162843448s) [noise=true , address=77.90.31.13:4634 , piece_id=NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.353007052 (632h31m28.224090097s) [noise=true , address=77.90.31.14:4673 , piece_id=FEZGA5U275AGSYP6L2DBFWTGCHC5S6Y4APJ7WGZ6NNTPP4E3QRBQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.386749041 (632h31m28.257832086s) [noise=true , address=45.91.116.13:4164 , piece_id=5NAIJXVZRI2ZK256AELB7M6Z4PZRELIL6H35EVOQXJK7444QWXAQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.459684066 (632h31m28.330767111s) [noise=true , address=145.79.227.33:4879 , piece_id=G3PEREJSNH3YF4KS2HYQO7GUFGTAA5ZKC2K5MVHPQO2EZNXBQSCQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.754224531 (632h31m28.625307576s) [noise=true , address=45.91.116.5:4319 , piece_id=OAJRVQHDC5T7C25HBIWZIWZXBFXPRHLAKUOC7RZUF6HI5N4PJINQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.012527725 (632h31m28.88361077s) [noise=true , address=145.79.227.30:4792 , piece_id=3PSRSL6JHSXPJYLM6CHKL4XDSROFLBVWTFYPL42SIPJQSPM6AJFQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.044467761 (632h31m28.915550806s) [noise=true , address=145.79.227.49:4372 , piece_id=3DNK6HF64TE4MU6M7AWTNW2ILTXNGNMPAX5HLKMROL77R5YJZIYA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.107424457 (632h31m28.978507502s) [noise=true , address=45.91.116.14:4197 , piece_id=YQSVWEYT4QAJPRK5FQUUF3U7IHUD2OTTMERFDD3XSJAM44XBKU6Q, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.132068652 (632h31m29.003151697s) [noise=true , address=145.79.227.34:4929 , piece_id=A7XSTYBGFCLEBSL7GU3N4RGPQB6MPEYJLZQQM7RIG4MEDRG5RJWQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.337855852 (632h31m29.208938887s) [noise=true , address=193.149.164.143:4698 , piece_id=JLTXGS4WB7RT7KA5DCX5XFFUZNXOMSR7EBBQJRLGZRJWKVRWQBOA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.337884647 (632h31m29.208967702s) [seg_pos={0 0}, upload=554251]: Segment reached optimal threshold of 14 pieces. Jan 1 00:00:29.338100017 (632h31m29.209183062s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF): protocol: write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection; write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection Jan 1 00:00:29.338108012 (632h31m29.209191057s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338168928 (632h31m29.209251993s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jan 1 00:00:29.338180740 (632h31m29.209263795s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338254290 (632h31m29.209337345s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jvSGi6s5yi55BBFJ4aErrap64wGP2WbDL9oUB6pdGiv8DM385): protocol: write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection; write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection Jan 1 00:00:29.338262516 (632h31m29.209345561s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338380130 (632h31m29.209463175s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection Jan 1 00:00:29.338387354 (632h31m29.209470399s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338772687 (632h31m29.209855732s) [seg_pos={0 0}, upload=554251]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:00:29.338880973 (632h31m29.209964019s) [upload=554251]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:00:29.338887636 (632h31m29.209970681s) [upload=554251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:00:29.640167090 (632h31m29.511250145s) [upload=554251]: Done uploading segments... Jan 1 00:00:29.640190766 (632h31m29.511273811s) [upload=554251]: Upload finished. err: <nil> Duration: 31.964697718s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 718.026864ms trace: https://jaeger.dev.storj.io/trace/6e3f973fa946f50a FAIL (2026-01-01 00:09:27 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52104->45.38.24.2:443: read: connection reset by peer Duration: 718.059696ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X....X...X............X....X....X......X....X....X.....XX] duration: 30.003045488s trace: https://jaeger.dev.storj.io/trace/b57c9ce3a641b25 FAIL (2026-01-01 00:09:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: EKPDGJZUQ4BREPSOBQOMTS4MRJCADWCEVVG6KPL23IK4JV2QVWRQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: GVQIXYHANL3YIRGUJQPHUJC3J726MDVQ3A6LCO3EZWY22JVTMVEQ) context deadline exceeded ======LOGS====== Jan 1 00:08:43.237629052 (632h39m43.108712087s) [upload=554374]: Starting upload Jan 1 00:08:43.237705036 (632h39m43.108788071s) [upload=554374]: Done starting upload Jan 1 00:08:43.237908544 (632h39m43.108991589s) [upload=554374]: Uploading segments... Jan 1 00:08:43.238053570 (632h39m43.109136605s) [upload=554374]: Got next segment. Inline: false Jan 1 00:08:43.238092534 (632h39m43.109175569s) [upload=554374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:08:43.238097122 (632h39m43.109180158s) [upload=554374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:08:43.486890530 (632h39m43.357973565s) [seg_pos={0 0}, upload=554374]: Begin upload segment... Jan 1 00:08:43.486947428 (632h39m43.358030473s) [seg_pos={0 0}, upload=554374]: Done begin upload segment. Jan 1 00:08:43.486979198 (632h39m43.358062233s) [noise=true , address=145.79.227.41:4086 , piece_id=YZ6IQM77DFOSLREK7OWKZV25RTG6Z65BODBRZLQMKCT24KRYNVUQ, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.486999718 (632h39m43.358082753s) [noise=true , address=145.79.227.22:4509 , piece_id=SHEQ2M7TP7KECAEQQ2JTPO7NLIUZXZUH6GZNSMI4XZZQZ7S7ZSAA, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.487678450 (632h39m43.358761485s) [noise=true , address=145.79.227.47:4267 , piece_id=C65WNMXNJ6R5BIBDT6FU7T4W37QPBSR42VNQTDLJ64LG5FCT3UHQ, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.488411496 (632h39m43.359494541s) [noise=true , address=145.79.227.8:4331 , piece_id=K7L6PUXWG2I3AO3FYE7OU5V7ZD5BABTX5XCPQA65Q5PCGYRQA7NQ, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.488695787 (632h39m43.359778822s) [noise=true , address=145.79.227.12:4767 , piece_id=QTJTJQU7LJPQXBPBFNJEF3W3CPLULYIWZIUS32YLPHYK52SYI3RA, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.488999505 (632h39m43.36008254s) [noise=true , address=145.79.227.16:4910 , piece_id=CQABRCU4GPUWKSGLV72QSTROQGAULASJGY6ORBVLBPZQFU7B5PQQ, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.489674530 (632h39m43.360757565s) [noise=true , address=145.79.227.43:4159 , piece_id=KBAKJ5YTGDNERJ3EMO5GW3OTD2QVGPOQ2HMJFPLUDGWP5MVYDQOA, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.490040848 (632h39m43.361123893s) [noise=true , address=145.79.227.38:4979 , piece_id=C6IFYES3CYLQV5VM2ODOW6XMLZ3N2EH5CSMCMUOSGGDQIGAGSHKQ, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.490963595 (632h39m43.36204664s) [noise=true , address=45.91.116.11:4120 , piece_id=EKPDGJZUQ4BREPSOBQOMTS4MRJCADWCEVVG6KPL23IK4JV2QVWRQ, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.491413161 (632h39m43.362496196s) [noise=true , address=145.79.227.11:4731 , piece_id=MOQQIH2MC3C3UOBMCI46V73MD44H4NXL2LUK4BGE2SNMVNCTOCVA, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.492108345 (632h39m43.36319138s) [noise=true , address=145.79.227.4:4186 , piece_id=2747ESKTYS5J6QGIYVD4VZKZDQWQ3PO3XDAOBLSHQY7EANIA5JRQ, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.492781146 (632h39m43.363864191s) [noise=true , address=145.79.227.6:4257 , piece_id=5PYYXVBQQW2M5HFXET7A4RSF5WCRPPJNSPXCTY7JIU3OA6RNYFIA, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.493786390 (632h39m43.364869425s) [noise=true , address=145.79.227.35:4481 , piece_id=VHBUMVG2I2CSLPHD4PJ765ANUUKF4HPO7VFHCKLFWQOT5ZZKQAUQ, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.494491713 (632h39m43.365574748s) [noise=true , address=45.91.116.7:4010 , piece_id=GVQIXYHANL3YIRGUJQPHUJC3J726MDVQ3A6LCO3EZWY22JVTMVEQ, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.494829045 (632h39m43.36591208s) [noise=true , address=77.90.31.21:4878 , piece_id=37JU57GYMDANWT2HTVAKPFL52KCG26UYZTVPVECSRFDC2WANVPCQ, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:43.496036815 (632h39m43.367119861s) [noise=true , address=145.79.227.50:4374 , piece_id=CFJUAFRUTCJB5IM6OFH47Z32PN2IR54VTKPGYHBWE2YW2REETQFA, seg_pos={0 0}, upload=554374]: Uploading piece... Jan 1 00:08:45.185886362 (632h39m45.056969397s) [upload=554374]: Next returned nil segment Jan 1 00:08:45.185903716 (632h39m45.056986751s) [upload=554374]: Waiting for error group managing segments... Jan 1 00:08:45.469997197 (632h39m45.341080232s) [noise=true , address=145.79.227.8:4331 , piece_id=K7L6PUXWG2I3AO3FYE7OU5V7ZD5BABTX5XCPQA65Q5PCGYRQA7NQ, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.477071337 (632h39m45.348154372s) [noise=true , address=145.79.227.47:4267 , piece_id=C65WNMXNJ6R5BIBDT6FU7T4W37QPBSR42VNQTDLJ64LG5FCT3UHQ, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.481529255 (632h39m45.3526123s) [noise=true , address=145.79.227.38:4979 , piece_id=C6IFYES3CYLQV5VM2ODOW6XMLZ3N2EH5CSMCMUOSGGDQIGAGSHKQ, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.505313472 (632h39m45.376396507s) [noise=true , address=145.79.227.12:4767 , piece_id=QTJTJQU7LJPQXBPBFNJEF3W3CPLULYIWZIUS32YLPHYK52SYI3RA, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.505872887 (632h39m45.376955923s) [noise=true , address=145.79.227.50:4374 , piece_id=CFJUAFRUTCJB5IM6OFH47Z32PN2IR54VTKPGYHBWE2YW2REETQFA, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.507171760 (632h39m45.378254796s) [noise=true , address=145.79.227.22:4509 , piece_id=SHEQ2M7TP7KECAEQQ2JTPO7NLIUZXZUH6GZNSMI4XZZQZ7S7ZSAA, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.510293565 (632h39m45.3813766s) [noise=true , address=145.79.227.11:4731 , piece_id=MOQQIH2MC3C3UOBMCI46V73MD44H4NXL2LUK4BGE2SNMVNCTOCVA, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.515957790 (632h39m45.387040835s) [noise=true , address=145.79.227.16:4910 , piece_id=CQABRCU4GPUWKSGLV72QSTROQGAULASJGY6ORBVLBPZQFU7B5PQQ, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.517315195 (632h39m45.38839824s) [noise=true , address=145.79.227.4:4186 , piece_id=2747ESKTYS5J6QGIYVD4VZKZDQWQ3PO3XDAOBLSHQY7EANIA5JRQ, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.602374606 (632h39m45.473457641s) [noise=true , address=145.79.227.6:4257 , piece_id=5PYYXVBQQW2M5HFXET7A4RSF5WCRPPJNSPXCTY7JIU3OA6RNYFIA, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.609542605 (632h39m45.48062564s) [noise=true , address=145.79.227.41:4086 , piece_id=YZ6IQM77DFOSLREK7OWKZV25RTG6Z65BODBRZLQMKCT24KRYNVUQ, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.885090004 (632h39m45.756173039s) [noise=true , address=45.91.116.11:4120 , piece_id=EKPDGJZUQ4BREPSOBQOMTS4MRJCADWCEVVG6KPL23IK4JV2QVWRQ, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.934104142 (632h39m45.805187177s) [noise=true , address=45.91.116.7:4010 , piece_id=GVQIXYHANL3YIRGUJQPHUJC3J726MDVQ3A6LCO3EZWY22JVTMVEQ, seg_pos={0 0}, upload=554374]: Done uploading piece. err: <nil> Jan 1 00:08:45.934133527 (632h39m45.805216572s) [seg_pos={0 0}, upload=554374]: Segment reached optimal threshold of 13 pieces. Jan 1 00:08:45.934366120 (632h39m45.805449165s) [noise=true , address=77.90.31.21:4878 , piece_id=37JU57GYMDANWT2HTVAKPFL52KCG26UYZTVPVECSRFDC2WANVPCQ, seg_pos={0 0}, upload=554374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:49272->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:49272->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49272->77.90.31.21:4878: use of closed network connection Jan 1 00:08:45.934376380 (632h39m45.805459415s) [noise=true , address=77.90.31.21:4878 , piece_id=37JU57GYMDANWT2HTVAKPFL52KCG26UYZTVPVECSRFDC2WANVPCQ, seg_pos={0 0}, upload=554374]: Long tail cancellation due to optimal threshold Jan 1 00:08:45.934611908 (632h39m45.805694953s) [noise=true , address=145.79.227.43:4159 , piece_id=KBAKJ5YTGDNERJ3EMO5GW3OTD2QVGPOQ2HMJFPLUDGWP5MVYDQOA, seg_pos={0 0}, upload=554374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Jan 1 00:08:45.934632047 (632h39m45.805715092s) [noise=true , address=145.79.227.43:4159 , piece_id=KBAKJ5YTGDNERJ3EMO5GW3OTD2QVGPOQ2HMJFPLUDGWP5MVYDQOA, seg_pos={0 0}, upload=554374]: Long tail cancellation due to optimal threshold Jan 1 00:08:45.934911169 (632h39m45.805994224s) [noise=true , address=145.79.227.35:4481 , piece_id=VHBUMVG2I2CSLPHD4PJ765ANUUKF4HPO7VFHCKLFWQOT5ZZKQAUQ, seg_pos={0 0}, upload=554374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 1 00:08:45.934921588 (632h39m45.806004623s) [noise=true , address=145.79.227.35:4481 , piece_id=VHBUMVG2I2CSLPHD4PJ765ANUUKF4HPO7VFHCKLFWQOT5ZZKQAUQ, seg_pos={0 0}, upload=554374]: Long tail cancellation due to optimal threshold Jan 1 00:08:45.934947548 (632h39m45.806030583s) [seg_pos={0 0}, upload=554374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:08:45.935131317 (632h39m45.806214373s) [upload=554374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:08:45.935139453 (632h39m45.806222508s) [upload=554374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:08:46.232283905 (632h39m46.10336696s) [upload=554374]: Done uploading segments... Jan 1 00:08:46.232308251 (632h39m46.103391296s) [upload=554374]: Upload finished. err: <nil> Duration: 30.003071887s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 738.446203ms trace: https://jaeger.dev.storj.io/trace/6e3f973fa946f50a FAIL (2026-01-01 00:09:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52104->45.38.24.2:443: read: connection reset by peer Duration: 738.482733ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..........................X....................X.........] duration: 31.518799801s trace: https://jaeger.dev.storj.io/trace/71cc346ea8b1cbd1 FAIL (2026-01-01 00:09:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, Piece ID: YQSVWEYT4QAJPRK5FQUUF3U7IHUD2OTTMERFDD3XSJAM44XBKU6Q) context deadline exceeded; piecestore: (Node ID: 1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, Piece ID: NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ) context deadline exceeded; piecestore: (Node ID: 12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof, Piece ID: OAJRVQHDC5T7C25HBIWZIWZXBFXPRHLAKUOC7RZUF6HI5N4PJINQ) context deadline exceeded ======LOGS====== Z4BAFLDDP62G2AEZ4QUDUUFYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.221779207 (632h31m26.092862242s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.221794527 (632h31m26.092877572s) [noise=true , address=145.79.227.43:4168 , piece_id=67E2PXGLNBVDVAWNMTIJ5NPIZ5KUFOSKOJVGMP7V5ULZQ2QO2HNA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222236768 (632h31m26.093319814s) [noise=true , address=145.79.227.19:4104 , piece_id=TRLT4BXD66MUVFYX7H35XP2PJJ3KN5F2HPHPN66L7KVK42ANCQYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222256006 (632h31m26.093339051s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222688048 (632h31m26.093771093s) [noise=true , address=77.90.31.13:4634 , piece_id=NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.223053204 (632h31m26.094136249s) [noise=true , address=145.79.227.49:4372 , piece_id=3DNK6HF64TE4MU6M7AWTNW2ILTXNGNMPAX5HLKMROL77R5YJZIYA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.223504623 (632h31m26.094587668s) [noise=true , address=77.90.31.14:4673 , piece_id=FEZGA5U275AGSYP6L2DBFWTGCHC5S6Y4APJ7WGZ6NNTPP4E3QRBQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224091681 (632h31m26.095174726s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224459571 (632h31m26.095542616s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224880994 (632h31m26.095964039s) [noise=true , address=45.91.116.13:4164 , piece_id=5NAIJXVZRI2ZK256AELB7M6Z4PZRELIL6H35EVOQXJK7444QWXAQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.225276838 (632h31m26.096359883s) [noise=true , address=193.149.164.143:4698 , piece_id=JLTXGS4WB7RT7KA5DCX5XFFUZNXOMSR7EBBQJRLGZRJWKVRWQBOA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.226050822 (632h31m26.097133897s) [noise=true , address=145.79.227.34:4929 , piece_id=A7XSTYBGFCLEBSL7GU3N4RGPQB6MPEYJLZQQM7RIG4MEDRG5RJWQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.228223077 (632h31m26.099306123s) [noise=true , address=145.79.227.33:4879 , piece_id=G3PEREJSNH3YF4KS2HYQO7GUFGTAA5ZKC2K5MVHPQO2EZNXBQSCQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:27.694323976 (632h31m27.565407021s) [upload=554251]: Next returned nil segment Jan 1 00:00:27.694348883 (632h31m27.565431928s) [upload=554251]: Waiting for error group managing segments... Jan 1 00:00:28.086794748 (632h31m27.957877793s) [noise=true , address=145.79.227.43:4168 , piece_id=67E2PXGLNBVDVAWNMTIJ5NPIZ5KUFOSKOJVGMP7V5ULZQ2QO2HNA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.105443428 (632h31m27.976526473s) [noise=true , address=145.79.227.19:4104 , piece_id=TRLT4BXD66MUVFYX7H35XP2PJJ3KN5F2HPHPN66L7KVK42ANCQYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.229480186 (632h31m28.100563231s) [noise=true , address=77.90.31.10:4529 , piece_id=SFBOVPQEXSGZ2H5YSXKIQO65NRCVIBOW3T6EAW6NUOXDS3A2X2VA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.273850481 (632h31m28.144933526s) [noise=true , address=64.49.13.154:4926 , piece_id=QGUG2VZT56RC22UM2ZGPLELFJTFZ4BAFLDDP62G2AEZ4QUDUUFYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.291760373 (632h31m28.162843448s) [noise=true , address=77.90.31.13:4634 , piece_id=NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.353007052 (632h31m28.224090097s) [noise=true , address=77.90.31.14:4673 , piece_id=FEZGA5U275AGSYP6L2DBFWTGCHC5S6Y4APJ7WGZ6NNTPP4E3QRBQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.386749041 (632h31m28.257832086s) [noise=true , address=45.91.116.13:4164 , piece_id=5NAIJXVZRI2ZK256AELB7M6Z4PZRELIL6H35EVOQXJK7444QWXAQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.459684066 (632h31m28.330767111s) [noise=true , address=145.79.227.33:4879 , piece_id=G3PEREJSNH3YF4KS2HYQO7GUFGTAA5ZKC2K5MVHPQO2EZNXBQSCQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.754224531 (632h31m28.625307576s) [noise=true , address=45.91.116.5:4319 , piece_id=OAJRVQHDC5T7C25HBIWZIWZXBFXPRHLAKUOC7RZUF6HI5N4PJINQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.012527725 (632h31m28.88361077s) [noise=true , address=145.79.227.30:4792 , piece_id=3PSRSL6JHSXPJYLM6CHKL4XDSROFLBVWTFYPL42SIPJQSPM6AJFQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.044467761 (632h31m28.915550806s) [noise=true , address=145.79.227.49:4372 , piece_id=3DNK6HF64TE4MU6M7AWTNW2ILTXNGNMPAX5HLKMROL77R5YJZIYA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.107424457 (632h31m28.978507502s) [noise=true , address=45.91.116.14:4197 , piece_id=YQSVWEYT4QAJPRK5FQUUF3U7IHUD2OTTMERFDD3XSJAM44XBKU6Q, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.132068652 (632h31m29.003151697s) [noise=true , address=145.79.227.34:4929 , piece_id=A7XSTYBGFCLEBSL7GU3N4RGPQB6MPEYJLZQQM7RIG4MEDRG5RJWQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.337855852 (632h31m29.208938887s) [noise=true , address=193.149.164.143:4698 , piece_id=JLTXGS4WB7RT7KA5DCX5XFFUZNXOMSR7EBBQJRLGZRJWKVRWQBOA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.337884647 (632h31m29.208967702s) [seg_pos={0 0}, upload=554251]: Segment reached optimal threshold of 14 pieces. Jan 1 00:00:29.338100017 (632h31m29.209183062s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF): protocol: write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection; write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection Jan 1 00:00:29.338108012 (632h31m29.209191057s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338168928 (632h31m29.209251993s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jan 1 00:00:29.338180740 (632h31m29.209263795s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338254290 (632h31m29.209337345s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jvSGi6s5yi55BBFJ4aErrap64wGP2WbDL9oUB6pdGiv8DM385): protocol: write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection; write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection Jan 1 00:00:29.338262516 (632h31m29.209345561s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338380130 (632h31m29.209463175s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection Jan 1 00:00:29.338387354 (632h31m29.209470399s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338772687 (632h31m29.209855732s) [seg_pos={0 0}, upload=554251]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:00:29.338880973 (632h31m29.209964019s) [upload=554251]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:00:29.338887636 (632h31m29.209970681s) [upload=554251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:00:29.640167090 (632h31m29.511250145s) [upload=554251]: Done uploading segments... Jan 1 00:00:29.640190766 (632h31m29.511273811s) [upload=554251]: Upload finished. err: <nil> Duration: 31.518843625s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X....X...X............X....X....X......X....X....X.....X] duration: 30.003850436s trace: https://jaeger.dev.storj.io/trace/7abbad41d8a9e1dc FAIL (2026-01-01 00:08:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: J2Y67WVAOTGZSUVBEK6MZVMVR4VZVSW4QDMFCUFRATICXTA5VT2A) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: M2YJP3BA4MW45FVLRNHSJTQ2ZYLX566I73DONWZBK25UVU2DPCGA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: NVPAFWQKWVIG3QKV4TIUKSRXL2EWZZZATYGNGLKRKLIVCBOL2HVQ) context deadline exceeded ======LOGS====== 2h39m5.945429332s) [upload=554368]: Uploading segments... Jan 1 00:08:06.074449253 (632h39m5.945532288s) [upload=554368]: Got next segment. Inline: false Jan 1 00:08:06.074475243 (632h39m5.945558278s) [upload=554368]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:08:06.074479541 (632h39m5.945562586s) [upload=554368]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:08:06.319372041 (632h39m6.190455076s) [seg_pos={0 0}, upload=554368]: Begin upload segment... Jan 1 00:08:06.319418479 (632h39m6.190501514s) [seg_pos={0 0}, upload=554368]: Done begin upload segment. Jan 1 00:08:06.319458836 (632h39m6.190541871s) [noise=true , address=45.91.116.11:4120 , piece_id=M2YJP3BA4MW45FVLRNHSJTQ2ZYLX566I73DONWZBK25UVU2DPCGA, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.322047715 (632h39m6.19313075s) [noise=true , address=145.79.227.20:4438 , piece_id=JZ7ZCN5I5RSY375SNDYQXLHQH7CWRY6GB2II2BXNNQTKWD3QTIBQ, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.322437798 (632h39m6.193520844s) [noise=true , address=45.91.116.10:4083 , piece_id=NAF4NCFNRHC46XEM2S6AAYJNTLL7S4ZTEJX3TBR5CSFJMI6I5LRA, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.322916640 (632h39m6.193999715s) [noise=true , address=45.91.116.14:4191 , piece_id=UW6QSA75W5WAAB5GFXSF4DCV5Z3NZVDTMINOPWMPAU442J2ODWTA, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.323071415 (632h39m6.19415445s) [noise=true , address=77.90.31.21:4877 , piece_id=PWK2FIIZ6JSXV33A5LVNZZC6MKLWHN54NAS3I4XN2IPZK7KQ4VOQ, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.323469232 (632h39m6.194552267s) [noise=true , address=45.91.116.12:4225 , piece_id=NVPAFWQKWVIG3QKV4TIUKSRXL2EWZZZATYGNGLKRKLIVCBOL2HVQ, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.323581215 (632h39m6.19466425s) [noise=true , address=145.79.227.2:4114 , piece_id=VEOSKMZ6IBWF5RLLILABSM5YAPUPVK7G5NOWDFNUP4OMNN65WNMA, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.323794641 (632h39m6.194877676s) [noise=true , address=145.79.227.15:4875 , piece_id=BFYKF244OSWW4SQ3R47D2NOZ32JJXTFHODO2PHUWYBONCYPATQGQ, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.323893409 (632h39m6.194976454s) [noise=true , address=45.91.116.13:4153 , piece_id=GSFYD7MALSAHH72UP5RSJQ352LH6FUD2YNC6HWR2WQH6X7PGXH6A, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.324211455 (632h39m6.1952945s) [noise=true , address=45.91.116.5:4300 , piece_id=KVRILKRXHPEFM5XNN7AYB4DGNQOKU5V7JEWPLUTSFYICFF7OYRJQ, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.324298852 (632h39m6.195381887s) [noise=true , address=145.79.227.50:4373 , piece_id=N2MAFMZ75DKHNHRRBICTGVNNSUQ5ZAIFWKT5LOBCACB3DIWCDWWQ, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.324555410 (632h39m6.195638455s) [noise=true , address=145.79.227.8:4331 , piece_id=27L2N6JNQX6KWUTK6XO2WHGWPC62BL4WBKDKLATXTZ5RFDXIFTKA, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.324613882 (632h39m6.195696927s) [noise=true , address=77.90.31.17:4770 , piece_id=QBN5FENGGE4VDU2WWKZ567K2MUOQMPLYWJJ57LIQDGQMY3S5JEIA, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.324846836 (632h39m6.195929871s) [noise=true , address=77.90.31.19:4844 , piece_id=J2Y67WVAOTGZSUVBEK6MZVMVR4VZVSW4QDMFCUFRATICXTA5VT2A, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.324939371 (632h39m6.196022406s) [noise=true , address=45.91.116.4:4937 , piece_id=5JCBTVG36C5BBC7X4UE7VTLH46Q4ANWOPK3SA7QW67GRBM2W2ILA, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:06.325257497 (632h39m6.196340532s) [noise=true , address=77.90.31.14:4661 , piece_id=T5IA3277ATREVFGXMPT5QQXSQ5M5U2IZVEMLK2OOONOD5WCB33JA, seg_pos={0 0}, upload=554368]: Uploading piece... Jan 1 00:08:08.032996608 (632h39m7.904079643s) [upload=554368]: Next returned nil segment Jan 1 00:08:08.033008792 (632h39m7.904091827s) [upload=554368]: Waiting for error group managing segments... Jan 1 00:08:08.312422031 (632h39m8.183505067s) [noise=true , address=145.79.227.8:4331 , piece_id=27L2N6JNQX6KWUTK6XO2WHGWPC62BL4WBKDKLATXTZ5RFDXIFTKA, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.315552793 (632h39m8.186635838s) [noise=true , address=145.79.227.2:4114 , piece_id=VEOSKMZ6IBWF5RLLILABSM5YAPUPVK7G5NOWDFNUP4OMNN65WNMA, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.329187064 (632h39m8.200270099s) [noise=true , address=145.79.227.50:4373 , piece_id=N2MAFMZ75DKHNHRRBICTGVNNSUQ5ZAIFWKT5LOBCACB3DIWCDWWQ, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.390396313 (632h39m8.261479348s) [noise=true , address=145.79.227.15:4875 , piece_id=BFYKF244OSWW4SQ3R47D2NOZ32JJXTFHODO2PHUWYBONCYPATQGQ, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.396360108 (632h39m8.267443153s) [noise=true , address=145.79.227.20:4438 , piece_id=JZ7ZCN5I5RSY375SNDYQXLHQH7CWRY6GB2II2BXNNQTKWD3QTIBQ, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.564318868 (632h39m8.435401903s) [noise=true , address=77.90.31.17:4770 , piece_id=QBN5FENGGE4VDU2WWKZ567K2MUOQMPLYWJJ57LIQDGQMY3S5JEIA, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.615389091 (632h39m8.486472136s) [noise=true , address=77.90.31.19:4844 , piece_id=J2Y67WVAOTGZSUVBEK6MZVMVR4VZVSW4QDMFCUFRATICXTA5VT2A, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.616727420 (632h39m8.487810455s) [noise=true , address=77.90.31.14:4661 , piece_id=T5IA3277ATREVFGXMPT5QQXSQ5M5U2IZVEMLK2OOONOD5WCB33JA, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.641585572 (632h39m8.512668617s) [noise=true , address=45.91.116.12:4225 , piece_id=NVPAFWQKWVIG3QKV4TIUKSRXL2EWZZZATYGNGLKRKLIVCBOL2HVQ, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.650458376 (632h39m8.521541421s) [noise=true , address=45.91.116.13:4153 , piece_id=GSFYD7MALSAHH72UP5RSJQ352LH6FUD2YNC6HWR2WQH6X7PGXH6A, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.663966467 (632h39m8.535049502s) [noise=true , address=77.90.31.21:4877 , piece_id=PWK2FIIZ6JSXV33A5LVNZZC6MKLWHN54NAS3I4XN2IPZK7KQ4VOQ, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.728445221 (632h39m8.599528266s) [noise=true , address=45.91.116.11:4120 , piece_id=M2YJP3BA4MW45FVLRNHSJTQ2ZYLX566I73DONWZBK25UVU2DPCGA, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.761099929 (632h39m8.632182964s) [noise=true , address=45.91.116.14:4191 , piece_id=UW6QSA75W5WAAB5GFXSF4DCV5Z3NZVDTMINOPWMPAU442J2ODWTA, seg_pos={0 0}, upload=554368]: Done uploading piece. err: <nil> Jan 1 00:08:08.761121250 (632h39m8.632204295s) [seg_pos={0 0}, upload=554368]: Segment reached optimal threshold of 13 pieces. Jan 1 00:08:08.761297325 (632h39m8.63238037s) [noise=true , address=45.91.116.5:4300 , piece_id=KVRILKRXHPEFM5XNN7AYB4DGNQOKU5V7JEWPLUTSFYICFF7OYRJQ, seg_pos={0 0}, upload=554368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50586->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50586->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->45.91.116.5:4300: use of closed network connection Jan 1 00:08:08.761303146 (632h39m8.632386181s) [noise=true , address=45.91.116.5:4300 , piece_id=KVRILKRXHPEFM5XNN7AYB4DGNQOKU5V7JEWPLUTSFYICFF7OYRJQ, seg_pos={0 0}, upload=554368]: Long tail cancellation due to optimal threshold Jan 1 00:08:08.761448873 (632h39m8.632531918s) [noise=true , address=45.91.116.10:4083 , piece_id=NAF4NCFNRHC46XEM2S6AAYJNTLL7S4ZTEJX3TBR5CSFJMI6I5LRA, seg_pos={0 0}, upload=554368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35560->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35560->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35560->45.91.116.10:4083: use of closed network connection Jan 1 00:08:08.761455726 (632h39m8.632538761s) [noise=true , address=45.91.116.10:4083 , piece_id=NAF4NCFNRHC46XEM2S6AAYJNTLL7S4ZTEJX3TBR5CSFJMI6I5LRA, seg_pos={0 0}, upload=554368]: Long tail cancellation due to optimal threshold Jan 1 00:08:08.761485453 (632h39m8.632568488s) [noise=true , address=45.91.116.4:4937 , piece_id=5JCBTVG36C5BBC7X4UE7VTLH46Q4ANWOPK3SA7QW67GRBM2W2ILA, seg_pos={0 0}, upload=554368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Jan 1 00:08:08.761493899 (632h39m8.632576944s) [noise=true , address=45.91.116.4:4937 , piece_id=5JCBTVG36C5BBC7X4UE7VTLH46Q4ANWOPK3SA7QW67GRBM2W2ILA, seg_pos={0 0}, upload=554368]: Long tail cancellation due to optimal threshold Jan 1 00:08:08.761537271 (632h39m8.632620306s) [seg_pos={0 0}, upload=554368]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:08:08.761596544 (632h39m8.632679579s) [upload=554368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:08:08.761600732 (632h39m8.632683767s) [upload=554368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:08:09.052630902 (632h39m8.923713937s) [upload=554368]: Done uploading segments... Jan 1 00:08:09.052649908 (632h39m8.923732943s) [upload=554368]: Upload finished. err: <nil> Duration: 30.003871897s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 740.608028ms trace: https://jaeger.dev.storj.io/trace/6e3f973fa946f50a FAIL (2026-01-01 00:08:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52104->45.38.24.2:443: read: connection reset by peer Duration: 740.638628ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..........................X....................X........] duration: 15.112702342s trace: https://jaeger.dev.storj.io/trace/71cc346ea8b1cbd1 FAIL (2026-01-01 00:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, Piece ID: YQSVWEYT4QAJPRK5FQUUF3U7IHUD2OTTMERFDD3XSJAM44XBKU6Q) context deadline exceeded; piecestore: (Node ID: 1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, Piece ID: NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ) context deadline exceeded; piecestore: (Node ID: 12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof, Piece ID: OAJRVQHDC5T7C25HBIWZIWZXBFXPRHLAKUOC7RZUF6HI5N4PJINQ) context deadline exceeded ======LOGS====== Z4BAFLDDP62G2AEZ4QUDUUFYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.221779207 (632h31m26.092862242s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.221794527 (632h31m26.092877572s) [noise=true , address=145.79.227.43:4168 , piece_id=67E2PXGLNBVDVAWNMTIJ5NPIZ5KUFOSKOJVGMP7V5ULZQ2QO2HNA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222236768 (632h31m26.093319814s) [noise=true , address=145.79.227.19:4104 , piece_id=TRLT4BXD66MUVFYX7H35XP2PJJ3KN5F2HPHPN66L7KVK42ANCQYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222256006 (632h31m26.093339051s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222688048 (632h31m26.093771093s) [noise=true , address=77.90.31.13:4634 , piece_id=NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.223053204 (632h31m26.094136249s) [noise=true , address=145.79.227.49:4372 , piece_id=3DNK6HF64TE4MU6M7AWTNW2ILTXNGNMPAX5HLKMROL77R5YJZIYA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.223504623 (632h31m26.094587668s) [noise=true , address=77.90.31.14:4673 , piece_id=FEZGA5U275AGSYP6L2DBFWTGCHC5S6Y4APJ7WGZ6NNTPP4E3QRBQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224091681 (632h31m26.095174726s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224459571 (632h31m26.095542616s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224880994 (632h31m26.095964039s) [noise=true , address=45.91.116.13:4164 , piece_id=5NAIJXVZRI2ZK256AELB7M6Z4PZRELIL6H35EVOQXJK7444QWXAQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.225276838 (632h31m26.096359883s) [noise=true , address=193.149.164.143:4698 , piece_id=JLTXGS4WB7RT7KA5DCX5XFFUZNXOMSR7EBBQJRLGZRJWKVRWQBOA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.226050822 (632h31m26.097133897s) [noise=true , address=145.79.227.34:4929 , piece_id=A7XSTYBGFCLEBSL7GU3N4RGPQB6MPEYJLZQQM7RIG4MEDRG5RJWQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.228223077 (632h31m26.099306123s) [noise=true , address=145.79.227.33:4879 , piece_id=G3PEREJSNH3YF4KS2HYQO7GUFGTAA5ZKC2K5MVHPQO2EZNXBQSCQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:27.694323976 (632h31m27.565407021s) [upload=554251]: Next returned nil segment Jan 1 00:00:27.694348883 (632h31m27.565431928s) [upload=554251]: Waiting for error group managing segments... Jan 1 00:00:28.086794748 (632h31m27.957877793s) [noise=true , address=145.79.227.43:4168 , piece_id=67E2PXGLNBVDVAWNMTIJ5NPIZ5KUFOSKOJVGMP7V5ULZQ2QO2HNA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.105443428 (632h31m27.976526473s) [noise=true , address=145.79.227.19:4104 , piece_id=TRLT4BXD66MUVFYX7H35XP2PJJ3KN5F2HPHPN66L7KVK42ANCQYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.229480186 (632h31m28.100563231s) [noise=true , address=77.90.31.10:4529 , piece_id=SFBOVPQEXSGZ2H5YSXKIQO65NRCVIBOW3T6EAW6NUOXDS3A2X2VA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.273850481 (632h31m28.144933526s) [noise=true , address=64.49.13.154:4926 , piece_id=QGUG2VZT56RC22UM2ZGPLELFJTFZ4BAFLDDP62G2AEZ4QUDUUFYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.291760373 (632h31m28.162843448s) [noise=true , address=77.90.31.13:4634 , piece_id=NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.353007052 (632h31m28.224090097s) [noise=true , address=77.90.31.14:4673 , piece_id=FEZGA5U275AGSYP6L2DBFWTGCHC5S6Y4APJ7WGZ6NNTPP4E3QRBQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.386749041 (632h31m28.257832086s) [noise=true , address=45.91.116.13:4164 , piece_id=5NAIJXVZRI2ZK256AELB7M6Z4PZRELIL6H35EVOQXJK7444QWXAQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.459684066 (632h31m28.330767111s) [noise=true , address=145.79.227.33:4879 , piece_id=G3PEREJSNH3YF4KS2HYQO7GUFGTAA5ZKC2K5MVHPQO2EZNXBQSCQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.754224531 (632h31m28.625307576s) [noise=true , address=45.91.116.5:4319 , piece_id=OAJRVQHDC5T7C25HBIWZIWZXBFXPRHLAKUOC7RZUF6HI5N4PJINQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.012527725 (632h31m28.88361077s) [noise=true , address=145.79.227.30:4792 , piece_id=3PSRSL6JHSXPJYLM6CHKL4XDSROFLBVWTFYPL42SIPJQSPM6AJFQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.044467761 (632h31m28.915550806s) [noise=true , address=145.79.227.49:4372 , piece_id=3DNK6HF64TE4MU6M7AWTNW2ILTXNGNMPAX5HLKMROL77R5YJZIYA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.107424457 (632h31m28.978507502s) [noise=true , address=45.91.116.14:4197 , piece_id=YQSVWEYT4QAJPRK5FQUUF3U7IHUD2OTTMERFDD3XSJAM44XBKU6Q, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.132068652 (632h31m29.003151697s) [noise=true , address=145.79.227.34:4929 , piece_id=A7XSTYBGFCLEBSL7GU3N4RGPQB6MPEYJLZQQM7RIG4MEDRG5RJWQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.337855852 (632h31m29.208938887s) [noise=true , address=193.149.164.143:4698 , piece_id=JLTXGS4WB7RT7KA5DCX5XFFUZNXOMSR7EBBQJRLGZRJWKVRWQBOA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.337884647 (632h31m29.208967702s) [seg_pos={0 0}, upload=554251]: Segment reached optimal threshold of 14 pieces. Jan 1 00:00:29.338100017 (632h31m29.209183062s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF): protocol: write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection; write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection Jan 1 00:00:29.338108012 (632h31m29.209191057s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338168928 (632h31m29.209251993s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jan 1 00:00:29.338180740 (632h31m29.209263795s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338254290 (632h31m29.209337345s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jvSGi6s5yi55BBFJ4aErrap64wGP2WbDL9oUB6pdGiv8DM385): protocol: write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection; write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection Jan 1 00:00:29.338262516 (632h31m29.209345561s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338380130 (632h31m29.209463175s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection Jan 1 00:00:29.338387354 (632h31m29.209470399s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338772687 (632h31m29.209855732s) [seg_pos={0 0}, upload=554251]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:00:29.338880973 (632h31m29.209964019s) [upload=554251]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:00:29.338887636 (632h31m29.209970681s) [upload=554251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:00:29.640167090 (632h31m29.511250145s) [upload=554251]: Done uploading segments... Jan 1 00:00:29.640190766 (632h31m29.511273811s) [upload=554251]: Upload finished. err: <nil> Duration: 15.112730977s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 716.545412ms trace: https://jaeger.dev.storj.io/trace/6e3f973fa946f50a FAIL (2026-01-01 00:07:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52104->45.38.24.2:443: read: connection reset by peer Duration: 716.567645ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X....X...X............X....X....X......X....X....X.....] duration: 6.273797827s trace: https://jaeger.dev.storj.io/trace/62565d86aa452b27 FAIL (2026-01-01 00:07:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA) context deadline exceeded ======LOGS====== 0:03:13.215173797 (632h34m13.086256842s) [upload=554293]: Starting upload Jan 1 00:03:13.215251625 (632h34m13.08633467s) [upload=554293]: Done starting upload Jan 1 00:03:13.215489879 (632h34m13.086572914s) [upload=554293]: Uploading segments... Jan 1 00:03:13.215621920 (632h34m13.086704965s) [upload=554293]: Got next segment. Inline: false Jan 1 00:03:13.215655564 (632h34m13.086738609s) [upload=554293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:03:13.215660954 (632h34m13.08674399s) [upload=554293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:03:13.466443208 (632h34m13.337526253s) [seg_pos={0 0}, upload=554293]: Begin upload segment... Jan 1 00:03:13.466494185 (632h34m13.33757723s) [seg_pos={0 0}, upload=554293]: Done begin upload segment. Jan 1 00:03:13.466523170 (632h34m13.337606205s) [noise=true , address=145.79.227.50:4375 , piece_id=RQ3D5SXEDIRBS7YA56TFZ7GZVL7IJ3WVURG7YI66OHLGDXKGFZEQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.466724423 (632h34m13.337807459s) [noise=true , address=45.91.116.11:4119 , piece_id=3AK2SKNIZQEDMQ63G4JEEJFYYS25PGOKXNBTIEWIKZSFR6H7SVMA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.467293877 (632h34m13.338376913s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.467654545 (632h34m13.33873758s) [noise=true , address=145.79.227.26:4653 , piece_id=KFJWX4SOTM5DLBL2JES4E7C35WJSHOFAUQWQFMWUFXELSSHOURCA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468103910 (632h34m13.339186945s) [noise=true , address=45.91.116.6:4974 , piece_id=W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468932307 (632h34m13.340015352s) [noise=true , address=145.79.227.16:4909 , piece_id=WHKIVTMIMH5ASWB3RDXC6HBPLA2DN7ZQAYDW3BDTIE6CKAIKHRZA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468996620 (632h34m13.340079655s) [noise=true , address=145.79.227.48:4303 , piece_id=67C2ANKBDCZO7IESTNGKDAVHPEXDUE5ZHM5BY5Y43S7WY4QY3BTA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.469825729 (632h34m13.340908764s) [noise=true , address=77.90.31.11:4553 , piece_id=OXABXPZRS4F2MOAIOA6F7KH466Q6KV7YXZFPNZBMYO5QW2NM23VQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470258733 (632h34m13.341341768s) [noise=true , address=145.79.227.23:4547 , piece_id=CE27VGWXCZXVPG2MLZOM5R3UGNKPRSOUNXG5IFOQCS63ZLUXC64A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470407796 (632h34m13.341490842s) [noise=true , address=145.79.227.13:4804 , piece_id=S3HSLEV5NCKSS3FIPMCWVRD33TYYYUIK5DZIKY5G2DMTDU2Z5DGA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470667341 (632h34m13.341750386s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471033037 (632h34m13.342116072s) [noise=true , address=145.79.227.51:4410 , piece_id=OCJT5H5DXZPYJI5UKDKWPVMG44AB4TJAYQQBESZI4TDRPZNHBT3A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471146663 (632h34m13.342229708s) [noise=true , address=145.79.227.38:4977 , piece_id=QJF6PPEAQNZ6VLZYVN4VVGN6HAZZNREE6YNT432RVHKQHVCC3R4Q, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471805297 (632h34m13.342888332s) [noise=true , address=145.79.227.45:4230 , piece_id=UDLKH27REUXB3HJE7IIWACZKZTFIMLIQXKDLMUGARCKTZMXP5C6A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.472334625 (632h34m13.34341767s) [noise=true , address=77.90.31.21:4880 , piece_id=HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.472813066 (632h34m13.343896111s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:15.183745018 (632h34m15.054828063s) [upload=554293]: Next returned nil segment Jan 1 00:03:15.183763083 (632h34m15.054846128s) [upload=554293]: Waiting for error group managing segments... Jan 1 00:03:15.459426483 (632h34m15.330509528s) [noise=true , address=145.79.227.38:4977 , piece_id=QJF6PPEAQNZ6VLZYVN4VVGN6HAZZNREE6YNT432RVHKQHVCC3R4Q, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.473867008 (632h34m15.344950043s) [noise=true , address=145.79.227.51:4410 , piece_id=OCJT5H5DXZPYJI5UKDKWPVMG44AB4TJAYQQBESZI4TDRPZNHBT3A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.479483984 (632h34m15.350567029s) [noise=true , address=145.79.227.45:4230 , piece_id=UDLKH27REUXB3HJE7IIWACZKZTFIMLIQXKDLMUGARCKTZMXP5C6A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.480028941 (632h34m15.351111976s) [noise=true , address=145.79.227.48:4303 , piece_id=67C2ANKBDCZO7IESTNGKDAVHPEXDUE5ZHM5BY5Y43S7WY4QY3BTA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.482241253 (632h34m15.353324299s) [noise=true , address=145.79.227.23:4547 , piece_id=CE27VGWXCZXVPG2MLZOM5R3UGNKPRSOUNXG5IFOQCS63ZLUXC64A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.482795108 (632h34m15.353878143s) [noise=true , address=145.79.227.16:4909 , piece_id=WHKIVTMIMH5ASWB3RDXC6HBPLA2DN7ZQAYDW3BDTIE6CKAIKHRZA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.485544574 (632h34m15.356627619s) [noise=true , address=145.79.227.50:4375 , piece_id=RQ3D5SXEDIRBS7YA56TFZ7GZVL7IJ3WVURG7YI66OHLGDXKGFZEQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.516207299 (632h34m15.387290334s) [noise=true , address=145.79.227.13:4804 , piece_id=S3HSLEV5NCKSS3FIPMCWVRD33TYYYUIK5DZIKY5G2DMTDU2Z5DGA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.631365352 (632h34m15.502448387s) [noise=true , address=145.79.227.26:4653 , piece_id=KFJWX4SOTM5DLBL2JES4E7C35WJSHOFAUQWQFMWUFXELSSHOURCA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.652714673 (632h34m15.523797708s) [noise=true , address=77.90.31.21:4880 , piece_id=HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.745297400 (632h34m15.616380435s) [noise=true , address=45.91.116.11:4119 , piece_id=3AK2SKNIZQEDMQ63G4JEEJFYYS25PGOKXNBTIEWIKZSFR6H7SVMA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.749953406 (632h34m15.621036441s) [noise=true , address=77.90.31.11:4553 , piece_id=OXABXPZRS4F2MOAIOA6F7KH466Q6KV7YXZFPNZBMYO5QW2NM23VQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:16.166855959 (632h34m16.037939004s) [noise=true , address=45.91.116.6:4974 , piece_id=W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:16.166880727 (632h34m16.037963762s) [seg_pos={0 0}, upload=554293]: Segment reached optimal threshold of 13 pieces. Jan 1 00:03:16.167120633 (632h34m16.038203698s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection Jan 1 00:03:16.167130663 (632h34m16.038213698s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167319702 (632h34m16.038402767s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 1 00:03:16.167363646 (632h34m16.038446701s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167482202 (632h34m16.038565237s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 00:03:16.167491610 (632h34m16.038574645s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167579778 (632h34m16.038662813s) [seg_pos={0 0}, upload=554293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:03:16.167747627 (632h34m16.038830672s) [upload=554293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:03:16.167755753 (632h34m16.038838798s) [upload=554293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:03:16.496622667 (632h34m16.367705712s) [upload=554293]: Done uploading segments... Jan 1 00:03:16.496646994 (632h34m16.367730029s) [upload=554293]: Upload finished. err: <nil> Duration: 6.273829138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.....X....X...X............X....X....X......X....X....X....] duration: 25.694518249s trace: https://jaeger.dev.storj.io/trace/62565d86aa452b27 FAIL (2026-01-01 00:06:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA) context deadline exceeded ======LOGS====== 0:03:13.215173797 (632h34m13.086256842s) [upload=554293]: Starting upload Jan 1 00:03:13.215251625 (632h34m13.08633467s) [upload=554293]: Done starting upload Jan 1 00:03:13.215489879 (632h34m13.086572914s) [upload=554293]: Uploading segments... Jan 1 00:03:13.215621920 (632h34m13.086704965s) [upload=554293]: Got next segment. Inline: false Jan 1 00:03:13.215655564 (632h34m13.086738609s) [upload=554293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:03:13.215660954 (632h34m13.08674399s) [upload=554293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:03:13.466443208 (632h34m13.337526253s) [seg_pos={0 0}, upload=554293]: Begin upload segment... Jan 1 00:03:13.466494185 (632h34m13.33757723s) [seg_pos={0 0}, upload=554293]: Done begin upload segment. Jan 1 00:03:13.466523170 (632h34m13.337606205s) [noise=true , address=145.79.227.50:4375 , piece_id=RQ3D5SXEDIRBS7YA56TFZ7GZVL7IJ3WVURG7YI66OHLGDXKGFZEQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.466724423 (632h34m13.337807459s) [noise=true , address=45.91.116.11:4119 , piece_id=3AK2SKNIZQEDMQ63G4JEEJFYYS25PGOKXNBTIEWIKZSFR6H7SVMA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.467293877 (632h34m13.338376913s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.467654545 (632h34m13.33873758s) [noise=true , address=145.79.227.26:4653 , piece_id=KFJWX4SOTM5DLBL2JES4E7C35WJSHOFAUQWQFMWUFXELSSHOURCA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468103910 (632h34m13.339186945s) [noise=true , address=45.91.116.6:4974 , piece_id=W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468932307 (632h34m13.340015352s) [noise=true , address=145.79.227.16:4909 , piece_id=WHKIVTMIMH5ASWB3RDXC6HBPLA2DN7ZQAYDW3BDTIE6CKAIKHRZA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468996620 (632h34m13.340079655s) [noise=true , address=145.79.227.48:4303 , piece_id=67C2ANKBDCZO7IESTNGKDAVHPEXDUE5ZHM5BY5Y43S7WY4QY3BTA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.469825729 (632h34m13.340908764s) [noise=true , address=77.90.31.11:4553 , piece_id=OXABXPZRS4F2MOAIOA6F7KH466Q6KV7YXZFPNZBMYO5QW2NM23VQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470258733 (632h34m13.341341768s) [noise=true , address=145.79.227.23:4547 , piece_id=CE27VGWXCZXVPG2MLZOM5R3UGNKPRSOUNXG5IFOQCS63ZLUXC64A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470407796 (632h34m13.341490842s) [noise=true , address=145.79.227.13:4804 , piece_id=S3HSLEV5NCKSS3FIPMCWVRD33TYYYUIK5DZIKY5G2DMTDU2Z5DGA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470667341 (632h34m13.341750386s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471033037 (632h34m13.342116072s) [noise=true , address=145.79.227.51:4410 , piece_id=OCJT5H5DXZPYJI5UKDKWPVMG44AB4TJAYQQBESZI4TDRPZNHBT3A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471146663 (632h34m13.342229708s) [noise=true , address=145.79.227.38:4977 , piece_id=QJF6PPEAQNZ6VLZYVN4VVGN6HAZZNREE6YNT432RVHKQHVCC3R4Q, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471805297 (632h34m13.342888332s) [noise=true , address=145.79.227.45:4230 , piece_id=UDLKH27REUXB3HJE7IIWACZKZTFIMLIQXKDLMUGARCKTZMXP5C6A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.472334625 (632h34m13.34341767s) [noise=true , address=77.90.31.21:4880 , piece_id=HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.472813066 (632h34m13.343896111s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:15.183745018 (632h34m15.054828063s) [upload=554293]: Next returned nil segment Jan 1 00:03:15.183763083 (632h34m15.054846128s) [upload=554293]: Waiting for error group managing segments... Jan 1 00:03:15.459426483 (632h34m15.330509528s) [noise=true , address=145.79.227.38:4977 , piece_id=QJF6PPEAQNZ6VLZYVN4VVGN6HAZZNREE6YNT432RVHKQHVCC3R4Q, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.473867008 (632h34m15.344950043s) [noise=true , address=145.79.227.51:4410 , piece_id=OCJT5H5DXZPYJI5UKDKWPVMG44AB4TJAYQQBESZI4TDRPZNHBT3A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.479483984 (632h34m15.350567029s) [noise=true , address=145.79.227.45:4230 , piece_id=UDLKH27REUXB3HJE7IIWACZKZTFIMLIQXKDLMUGARCKTZMXP5C6A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.480028941 (632h34m15.351111976s) [noise=true , address=145.79.227.48:4303 , piece_id=67C2ANKBDCZO7IESTNGKDAVHPEXDUE5ZHM5BY5Y43S7WY4QY3BTA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.482241253 (632h34m15.353324299s) [noise=true , address=145.79.227.23:4547 , piece_id=CE27VGWXCZXVPG2MLZOM5R3UGNKPRSOUNXG5IFOQCS63ZLUXC64A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.482795108 (632h34m15.353878143s) [noise=true , address=145.79.227.16:4909 , piece_id=WHKIVTMIMH5ASWB3RDXC6HBPLA2DN7ZQAYDW3BDTIE6CKAIKHRZA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.485544574 (632h34m15.356627619s) [noise=true , address=145.79.227.50:4375 , piece_id=RQ3D5SXEDIRBS7YA56TFZ7GZVL7IJ3WVURG7YI66OHLGDXKGFZEQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.516207299 (632h34m15.387290334s) [noise=true , address=145.79.227.13:4804 , piece_id=S3HSLEV5NCKSS3FIPMCWVRD33TYYYUIK5DZIKY5G2DMTDU2Z5DGA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.631365352 (632h34m15.502448387s) [noise=true , address=145.79.227.26:4653 , piece_id=KFJWX4SOTM5DLBL2JES4E7C35WJSHOFAUQWQFMWUFXELSSHOURCA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.652714673 (632h34m15.523797708s) [noise=true , address=77.90.31.21:4880 , piece_id=HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.745297400 (632h34m15.616380435s) [noise=true , address=45.91.116.11:4119 , piece_id=3AK2SKNIZQEDMQ63G4JEEJFYYS25PGOKXNBTIEWIKZSFR6H7SVMA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.749953406 (632h34m15.621036441s) [noise=true , address=77.90.31.11:4553 , piece_id=OXABXPZRS4F2MOAIOA6F7KH466Q6KV7YXZFPNZBMYO5QW2NM23VQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:16.166855959 (632h34m16.037939004s) [noise=true , address=45.91.116.6:4974 , piece_id=W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:16.166880727 (632h34m16.037963762s) [seg_pos={0 0}, upload=554293]: Segment reached optimal threshold of 13 pieces. Jan 1 00:03:16.167120633 (632h34m16.038203698s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection Jan 1 00:03:16.167130663 (632h34m16.038213698s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167319702 (632h34m16.038402767s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 1 00:03:16.167363646 (632h34m16.038446701s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167482202 (632h34m16.038565237s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 00:03:16.167491610 (632h34m16.038574645s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167579778 (632h34m16.038662813s) [seg_pos={0 0}, upload=554293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:03:16.167747627 (632h34m16.038830672s) [upload=554293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:03:16.167755753 (632h34m16.038838798s) [upload=554293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:03:16.496622667 (632h34m16.367705712s) [upload=554293]: Done uploading segments... Jan 1 00:03:16.496646994 (632h34m16.367730029s) [upload=554293]: Upload finished. err: <nil> Duration: 25.694543928s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 711.0459ms trace: https://jaeger.dev.storj.io/trace/6e3f973fa946f50a FAIL (2026-01-01 00:06:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52104->45.38.24.2:443: read: connection reset by peer Duration: 711.0727ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...X..........................X....................X.......] duration: 14.847638057s trace: https://jaeger.dev.storj.io/trace/71cc346ea8b1cbd1 FAIL (2026-01-01 00:06:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, Piece ID: YQSVWEYT4QAJPRK5FQUUF3U7IHUD2OTTMERFDD3XSJAM44XBKU6Q) context deadline exceeded; piecestore: (Node ID: 1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, Piece ID: NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ) context deadline exceeded; piecestore: (Node ID: 12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof, Piece ID: OAJRVQHDC5T7C25HBIWZIWZXBFXPRHLAKUOC7RZUF6HI5N4PJINQ) context deadline exceeded ======LOGS====== Z4BAFLDDP62G2AEZ4QUDUUFYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.221779207 (632h31m26.092862242s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.221794527 (632h31m26.092877572s) [noise=true , address=145.79.227.43:4168 , piece_id=67E2PXGLNBVDVAWNMTIJ5NPIZ5KUFOSKOJVGMP7V5ULZQ2QO2HNA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222236768 (632h31m26.093319814s) [noise=true , address=145.79.227.19:4104 , piece_id=TRLT4BXD66MUVFYX7H35XP2PJJ3KN5F2HPHPN66L7KVK42ANCQYQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222256006 (632h31m26.093339051s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.222688048 (632h31m26.093771093s) [noise=true , address=77.90.31.13:4634 , piece_id=NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.223053204 (632h31m26.094136249s) [noise=true , address=145.79.227.49:4372 , piece_id=3DNK6HF64TE4MU6M7AWTNW2ILTXNGNMPAX5HLKMROL77R5YJZIYA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.223504623 (632h31m26.094587668s) [noise=true , address=77.90.31.14:4673 , piece_id=FEZGA5U275AGSYP6L2DBFWTGCHC5S6Y4APJ7WGZ6NNTPP4E3QRBQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224091681 (632h31m26.095174726s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224459571 (632h31m26.095542616s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.224880994 (632h31m26.095964039s) [noise=true , address=45.91.116.13:4164 , piece_id=5NAIJXVZRI2ZK256AELB7M6Z4PZRELIL6H35EVOQXJK7444QWXAQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.225276838 (632h31m26.096359883s) [noise=true , address=193.149.164.143:4698 , piece_id=JLTXGS4WB7RT7KA5DCX5XFFUZNXOMSR7EBBQJRLGZRJWKVRWQBOA, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.226050822 (632h31m26.097133897s) [noise=true , address=145.79.227.34:4929 , piece_id=A7XSTYBGFCLEBSL7GU3N4RGPQB6MPEYJLZQQM7RIG4MEDRG5RJWQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:26.228223077 (632h31m26.099306123s) [noise=true , address=145.79.227.33:4879 , piece_id=G3PEREJSNH3YF4KS2HYQO7GUFGTAA5ZKC2K5MVHPQO2EZNXBQSCQ, seg_pos={0 0}, upload=554251]: Uploading piece... Jan 1 00:00:27.694323976 (632h31m27.565407021s) [upload=554251]: Next returned nil segment Jan 1 00:00:27.694348883 (632h31m27.565431928s) [upload=554251]: Waiting for error group managing segments... Jan 1 00:00:28.086794748 (632h31m27.957877793s) [noise=true , address=145.79.227.43:4168 , piece_id=67E2PXGLNBVDVAWNMTIJ5NPIZ5KUFOSKOJVGMP7V5ULZQ2QO2HNA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.105443428 (632h31m27.976526473s) [noise=true , address=145.79.227.19:4104 , piece_id=TRLT4BXD66MUVFYX7H35XP2PJJ3KN5F2HPHPN66L7KVK42ANCQYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.229480186 (632h31m28.100563231s) [noise=true , address=77.90.31.10:4529 , piece_id=SFBOVPQEXSGZ2H5YSXKIQO65NRCVIBOW3T6EAW6NUOXDS3A2X2VA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.273850481 (632h31m28.144933526s) [noise=true , address=64.49.13.154:4926 , piece_id=QGUG2VZT56RC22UM2ZGPLELFJTFZ4BAFLDDP62G2AEZ4QUDUUFYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.291760373 (632h31m28.162843448s) [noise=true , address=77.90.31.13:4634 , piece_id=NS7GZJKKGU6N4IZZS4OJ226V6NA3RVXCES7QWJLNSCKDYVEURWQQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.353007052 (632h31m28.224090097s) [noise=true , address=77.90.31.14:4673 , piece_id=FEZGA5U275AGSYP6L2DBFWTGCHC5S6Y4APJ7WGZ6NNTPP4E3QRBQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.386749041 (632h31m28.257832086s) [noise=true , address=45.91.116.13:4164 , piece_id=5NAIJXVZRI2ZK256AELB7M6Z4PZRELIL6H35EVOQXJK7444QWXAQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.459684066 (632h31m28.330767111s) [noise=true , address=145.79.227.33:4879 , piece_id=G3PEREJSNH3YF4KS2HYQO7GUFGTAA5ZKC2K5MVHPQO2EZNXBQSCQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:28.754224531 (632h31m28.625307576s) [noise=true , address=45.91.116.5:4319 , piece_id=OAJRVQHDC5T7C25HBIWZIWZXBFXPRHLAKUOC7RZUF6HI5N4PJINQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.012527725 (632h31m28.88361077s) [noise=true , address=145.79.227.30:4792 , piece_id=3PSRSL6JHSXPJYLM6CHKL4XDSROFLBVWTFYPL42SIPJQSPM6AJFQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.044467761 (632h31m28.915550806s) [noise=true , address=145.79.227.49:4372 , piece_id=3DNK6HF64TE4MU6M7AWTNW2ILTXNGNMPAX5HLKMROL77R5YJZIYA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.107424457 (632h31m28.978507502s) [noise=true , address=45.91.116.14:4197 , piece_id=YQSVWEYT4QAJPRK5FQUUF3U7IHUD2OTTMERFDD3XSJAM44XBKU6Q, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.132068652 (632h31m29.003151697s) [noise=true , address=145.79.227.34:4929 , piece_id=A7XSTYBGFCLEBSL7GU3N4RGPQB6MPEYJLZQQM7RIG4MEDRG5RJWQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.337855852 (632h31m29.208938887s) [noise=true , address=193.149.164.143:4698 , piece_id=JLTXGS4WB7RT7KA5DCX5XFFUZNXOMSR7EBBQJRLGZRJWKVRWQBOA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: <nil> Jan 1 00:00:29.337884647 (632h31m29.208967702s) [seg_pos={0 0}, upload=554251]: Segment reached optimal threshold of 14 pieces. Jan 1 00:00:29.338100017 (632h31m29.209183062s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF): protocol: write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection; write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41586->145.79.227.3:4154: use of closed network connection Jan 1 00:00:29.338108012 (632h31m29.209191057s) [noise=true , address=145.79.227.3:4154 , piece_id=SFZHOB7CAR522SGI53ZRVJKW2HKZV3EPGIIU2SGN7PEEPCHI6PXQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338168928 (632h31m29.209251993s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jan 1 00:00:29.338180740 (632h31m29.209263795s) [noise=true , address=145.79.227.13:4834 , piece_id=OXXOHHS2THZULHWZ36743SELY5H3VHCJVA57WWZVS4EUWU5P5RHQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338254290 (632h31m29.209337345s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jvSGi6s5yi55BBFJ4aErrap64wGP2WbDL9oUB6pdGiv8DM385): protocol: write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection; write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50996->145.79.227.21:4486: use of closed network connection Jan 1 00:00:29.338262516 (632h31m29.209345561s) [noise=true , address=145.79.227.21:4486 , piece_id=EQ7Q2UFHRDBYUV5TJGADMJSTWCJBNWXCWTCKCJ5EYQS3HFFBTUUA, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338380130 (632h31m29.209463175s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35970->64.49.13.169:4707: use of closed network connection Jan 1 00:00:29.338387354 (632h31m29.209470399s) [noise=true , address=64.49.13.169:4707 , piece_id=VBFBGKO2Z62QLNJJLCYX56V7LA5NC77PT76VSBBVIMNNHR766SYQ, seg_pos={0 0}, upload=554251]: Long tail cancellation due to optimal threshold Jan 1 00:00:29.338772687 (632h31m29.209855732s) [seg_pos={0 0}, upload=554251]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:00:29.338880973 (632h31m29.209964019s) [upload=554251]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:00:29.338887636 (632h31m29.209970681s) [upload=554251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:00:29.640167090 (632h31m29.511250145s) [upload=554251]: Done uploading segments... Jan 1 00:00:29.640190766 (632h31m29.511273811s) [upload=554251]: Upload finished. err: <nil> Duration: 14.84766513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.....X....X...X............X....X....X......X....X....X...] duration: 8.437855806s trace: https://jaeger.dev.storj.io/trace/62565d86aa452b27 FAIL (2026-01-01 00:05:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA) context deadline exceeded ======LOGS====== 0:03:13.215173797 (632h34m13.086256842s) [upload=554293]: Starting upload Jan 1 00:03:13.215251625 (632h34m13.08633467s) [upload=554293]: Done starting upload Jan 1 00:03:13.215489879 (632h34m13.086572914s) [upload=554293]: Uploading segments... Jan 1 00:03:13.215621920 (632h34m13.086704965s) [upload=554293]: Got next segment. Inline: false Jan 1 00:03:13.215655564 (632h34m13.086738609s) [upload=554293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:03:13.215660954 (632h34m13.08674399s) [upload=554293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:03:13.466443208 (632h34m13.337526253s) [seg_pos={0 0}, upload=554293]: Begin upload segment... Jan 1 00:03:13.466494185 (632h34m13.33757723s) [seg_pos={0 0}, upload=554293]: Done begin upload segment. Jan 1 00:03:13.466523170 (632h34m13.337606205s) [noise=true , address=145.79.227.50:4375 , piece_id=RQ3D5SXEDIRBS7YA56TFZ7GZVL7IJ3WVURG7YI66OHLGDXKGFZEQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.466724423 (632h34m13.337807459s) [noise=true , address=45.91.116.11:4119 , piece_id=3AK2SKNIZQEDMQ63G4JEEJFYYS25PGOKXNBTIEWIKZSFR6H7SVMA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.467293877 (632h34m13.338376913s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.467654545 (632h34m13.33873758s) [noise=true , address=145.79.227.26:4653 , piece_id=KFJWX4SOTM5DLBL2JES4E7C35WJSHOFAUQWQFMWUFXELSSHOURCA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468103910 (632h34m13.339186945s) [noise=true , address=45.91.116.6:4974 , piece_id=W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468932307 (632h34m13.340015352s) [noise=true , address=145.79.227.16:4909 , piece_id=WHKIVTMIMH5ASWB3RDXC6HBPLA2DN7ZQAYDW3BDTIE6CKAIKHRZA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468996620 (632h34m13.340079655s) [noise=true , address=145.79.227.48:4303 , piece_id=67C2ANKBDCZO7IESTNGKDAVHPEXDUE5ZHM5BY5Y43S7WY4QY3BTA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.469825729 (632h34m13.340908764s) [noise=true , address=77.90.31.11:4553 , piece_id=OXABXPZRS4F2MOAIOA6F7KH466Q6KV7YXZFPNZBMYO5QW2NM23VQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470258733 (632h34m13.341341768s) [noise=true , address=145.79.227.23:4547 , piece_id=CE27VGWXCZXVPG2MLZOM5R3UGNKPRSOUNXG5IFOQCS63ZLUXC64A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470407796 (632h34m13.341490842s) [noise=true , address=145.79.227.13:4804 , piece_id=S3HSLEV5NCKSS3FIPMCWVRD33TYYYUIK5DZIKY5G2DMTDU2Z5DGA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470667341 (632h34m13.341750386s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471033037 (632h34m13.342116072s) [noise=true , address=145.79.227.51:4410 , piece_id=OCJT5H5DXZPYJI5UKDKWPVMG44AB4TJAYQQBESZI4TDRPZNHBT3A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471146663 (632h34m13.342229708s) [noise=true , address=145.79.227.38:4977 , piece_id=QJF6PPEAQNZ6VLZYVN4VVGN6HAZZNREE6YNT432RVHKQHVCC3R4Q, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471805297 (632h34m13.342888332s) [noise=true , address=145.79.227.45:4230 , piece_id=UDLKH27REUXB3HJE7IIWACZKZTFIMLIQXKDLMUGARCKTZMXP5C6A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.472334625 (632h34m13.34341767s) [noise=true , address=77.90.31.21:4880 , piece_id=HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.472813066 (632h34m13.343896111s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:15.183745018 (632h34m15.054828063s) [upload=554293]: Next returned nil segment Jan 1 00:03:15.183763083 (632h34m15.054846128s) [upload=554293]: Waiting for error group managing segments... Jan 1 00:03:15.459426483 (632h34m15.330509528s) [noise=true , address=145.79.227.38:4977 , piece_id=QJF6PPEAQNZ6VLZYVN4VVGN6HAZZNREE6YNT432RVHKQHVCC3R4Q, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.473867008 (632h34m15.344950043s) [noise=true , address=145.79.227.51:4410 , piece_id=OCJT5H5DXZPYJI5UKDKWPVMG44AB4TJAYQQBESZI4TDRPZNHBT3A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.479483984 (632h34m15.350567029s) [noise=true , address=145.79.227.45:4230 , piece_id=UDLKH27REUXB3HJE7IIWACZKZTFIMLIQXKDLMUGARCKTZMXP5C6A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.480028941 (632h34m15.351111976s) [noise=true , address=145.79.227.48:4303 , piece_id=67C2ANKBDCZO7IESTNGKDAVHPEXDUE5ZHM5BY5Y43S7WY4QY3BTA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.482241253 (632h34m15.353324299s) [noise=true , address=145.79.227.23:4547 , piece_id=CE27VGWXCZXVPG2MLZOM5R3UGNKPRSOUNXG5IFOQCS63ZLUXC64A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.482795108 (632h34m15.353878143s) [noise=true , address=145.79.227.16:4909 , piece_id=WHKIVTMIMH5ASWB3RDXC6HBPLA2DN7ZQAYDW3BDTIE6CKAIKHRZA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.485544574 (632h34m15.356627619s) [noise=true , address=145.79.227.50:4375 , piece_id=RQ3D5SXEDIRBS7YA56TFZ7GZVL7IJ3WVURG7YI66OHLGDXKGFZEQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.516207299 (632h34m15.387290334s) [noise=true , address=145.79.227.13:4804 , piece_id=S3HSLEV5NCKSS3FIPMCWVRD33TYYYUIK5DZIKY5G2DMTDU2Z5DGA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.631365352 (632h34m15.502448387s) [noise=true , address=145.79.227.26:4653 , piece_id=KFJWX4SOTM5DLBL2JES4E7C35WJSHOFAUQWQFMWUFXELSSHOURCA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.652714673 (632h34m15.523797708s) [noise=true , address=77.90.31.21:4880 , piece_id=HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.745297400 (632h34m15.616380435s) [noise=true , address=45.91.116.11:4119 , piece_id=3AK2SKNIZQEDMQ63G4JEEJFYYS25PGOKXNBTIEWIKZSFR6H7SVMA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.749953406 (632h34m15.621036441s) [noise=true , address=77.90.31.11:4553 , piece_id=OXABXPZRS4F2MOAIOA6F7KH466Q6KV7YXZFPNZBMYO5QW2NM23VQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:16.166855959 (632h34m16.037939004s) [noise=true , address=45.91.116.6:4974 , piece_id=W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:16.166880727 (632h34m16.037963762s) [seg_pos={0 0}, upload=554293]: Segment reached optimal threshold of 13 pieces. Jan 1 00:03:16.167120633 (632h34m16.038203698s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection Jan 1 00:03:16.167130663 (632h34m16.038213698s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167319702 (632h34m16.038402767s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 1 00:03:16.167363646 (632h34m16.038446701s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167482202 (632h34m16.038565237s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 00:03:16.167491610 (632h34m16.038574645s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167579778 (632h34m16.038662813s) [seg_pos={0 0}, upload=554293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:03:16.167747627 (632h34m16.038830672s) [upload=554293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:03:16.167755753 (632h34m16.038838798s) [upload=554293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:03:16.496622667 (632h34m16.367705712s) [upload=554293]: Done uploading segments... Jan 1 00:03:16.496646994 (632h34m16.367730029s) [upload=554293]: Upload finished. err: <nil> Duration: 8.437883438s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 771.745159ms trace: https://jaeger.dev.storj.io/trace/6e3f973fa946f50a FAIL (2026-01-01 00:05:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:52104->45.38.24.2:443: read: connection reset by peer Duration: 771.769844ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.....X....X...X............X....X....X......X....X....X..] duration: 22.727286717s trace: https://jaeger.dev.storj.io/trace/62565d86aa452b27 FAIL (2026-01-01 00:05:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA) context deadline exceeded ======LOGS====== 0:03:13.215173797 (632h34m13.086256842s) [upload=554293]: Starting upload Jan 1 00:03:13.215251625 (632h34m13.08633467s) [upload=554293]: Done starting upload Jan 1 00:03:13.215489879 (632h34m13.086572914s) [upload=554293]: Uploading segments... Jan 1 00:03:13.215621920 (632h34m13.086704965s) [upload=554293]: Got next segment. Inline: false Jan 1 00:03:13.215655564 (632h34m13.086738609s) [upload=554293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:03:13.215660954 (632h34m13.08674399s) [upload=554293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:03:13.466443208 (632h34m13.337526253s) [seg_pos={0 0}, upload=554293]: Begin upload segment... Jan 1 00:03:13.466494185 (632h34m13.33757723s) [seg_pos={0 0}, upload=554293]: Done begin upload segment. Jan 1 00:03:13.466523170 (632h34m13.337606205s) [noise=true , address=145.79.227.50:4375 , piece_id=RQ3D5SXEDIRBS7YA56TFZ7GZVL7IJ3WVURG7YI66OHLGDXKGFZEQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.466724423 (632h34m13.337807459s) [noise=true , address=45.91.116.11:4119 , piece_id=3AK2SKNIZQEDMQ63G4JEEJFYYS25PGOKXNBTIEWIKZSFR6H7SVMA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.467293877 (632h34m13.338376913s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.467654545 (632h34m13.33873758s) [noise=true , address=145.79.227.26:4653 , piece_id=KFJWX4SOTM5DLBL2JES4E7C35WJSHOFAUQWQFMWUFXELSSHOURCA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468103910 (632h34m13.339186945s) [noise=true , address=45.91.116.6:4974 , piece_id=W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468932307 (632h34m13.340015352s) [noise=true , address=145.79.227.16:4909 , piece_id=WHKIVTMIMH5ASWB3RDXC6HBPLA2DN7ZQAYDW3BDTIE6CKAIKHRZA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.468996620 (632h34m13.340079655s) [noise=true , address=145.79.227.48:4303 , piece_id=67C2ANKBDCZO7IESTNGKDAVHPEXDUE5ZHM5BY5Y43S7WY4QY3BTA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.469825729 (632h34m13.340908764s) [noise=true , address=77.90.31.11:4553 , piece_id=OXABXPZRS4F2MOAIOA6F7KH466Q6KV7YXZFPNZBMYO5QW2NM23VQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470258733 (632h34m13.341341768s) [noise=true , address=145.79.227.23:4547 , piece_id=CE27VGWXCZXVPG2MLZOM5R3UGNKPRSOUNXG5IFOQCS63ZLUXC64A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470407796 (632h34m13.341490842s) [noise=true , address=145.79.227.13:4804 , piece_id=S3HSLEV5NCKSS3FIPMCWVRD33TYYYUIK5DZIKY5G2DMTDU2Z5DGA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.470667341 (632h34m13.341750386s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471033037 (632h34m13.342116072s) [noise=true , address=145.79.227.51:4410 , piece_id=OCJT5H5DXZPYJI5UKDKWPVMG44AB4TJAYQQBESZI4TDRPZNHBT3A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471146663 (632h34m13.342229708s) [noise=true , address=145.79.227.38:4977 , piece_id=QJF6PPEAQNZ6VLZYVN4VVGN6HAZZNREE6YNT432RVHKQHVCC3R4Q, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.471805297 (632h34m13.342888332s) [noise=true , address=145.79.227.45:4230 , piece_id=UDLKH27REUXB3HJE7IIWACZKZTFIMLIQXKDLMUGARCKTZMXP5C6A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.472334625 (632h34m13.34341767s) [noise=true , address=77.90.31.21:4880 , piece_id=HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:13.472813066 (632h34m13.343896111s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Uploading piece... Jan 1 00:03:15.183745018 (632h34m15.054828063s) [upload=554293]: Next returned nil segment Jan 1 00:03:15.183763083 (632h34m15.054846128s) [upload=554293]: Waiting for error group managing segments... Jan 1 00:03:15.459426483 (632h34m15.330509528s) [noise=true , address=145.79.227.38:4977 , piece_id=QJF6PPEAQNZ6VLZYVN4VVGN6HAZZNREE6YNT432RVHKQHVCC3R4Q, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.473867008 (632h34m15.344950043s) [noise=true , address=145.79.227.51:4410 , piece_id=OCJT5H5DXZPYJI5UKDKWPVMG44AB4TJAYQQBESZI4TDRPZNHBT3A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.479483984 (632h34m15.350567029s) [noise=true , address=145.79.227.45:4230 , piece_id=UDLKH27REUXB3HJE7IIWACZKZTFIMLIQXKDLMUGARCKTZMXP5C6A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.480028941 (632h34m15.351111976s) [noise=true , address=145.79.227.48:4303 , piece_id=67C2ANKBDCZO7IESTNGKDAVHPEXDUE5ZHM5BY5Y43S7WY4QY3BTA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.482241253 (632h34m15.353324299s) [noise=true , address=145.79.227.23:4547 , piece_id=CE27VGWXCZXVPG2MLZOM5R3UGNKPRSOUNXG5IFOQCS63ZLUXC64A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.482795108 (632h34m15.353878143s) [noise=true , address=145.79.227.16:4909 , piece_id=WHKIVTMIMH5ASWB3RDXC6HBPLA2DN7ZQAYDW3BDTIE6CKAIKHRZA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.485544574 (632h34m15.356627619s) [noise=true , address=145.79.227.50:4375 , piece_id=RQ3D5SXEDIRBS7YA56TFZ7GZVL7IJ3WVURG7YI66OHLGDXKGFZEQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.516207299 (632h34m15.387290334s) [noise=true , address=145.79.227.13:4804 , piece_id=S3HSLEV5NCKSS3FIPMCWVRD33TYYYUIK5DZIKY5G2DMTDU2Z5DGA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.631365352 (632h34m15.502448387s) [noise=true , address=145.79.227.26:4653 , piece_id=KFJWX4SOTM5DLBL2JES4E7C35WJSHOFAUQWQFMWUFXELSSHOURCA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.652714673 (632h34m15.523797708s) [noise=true , address=77.90.31.21:4880 , piece_id=HYE6NKAWZKJA4L5GC4CQTAYODD7RG33IRLRNQOSCTTKEJ3NHESTQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.745297400 (632h34m15.616380435s) [noise=true , address=45.91.116.11:4119 , piece_id=3AK2SKNIZQEDMQ63G4JEEJFYYS25PGOKXNBTIEWIKZSFR6H7SVMA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:15.749953406 (632h34m15.621036441s) [noise=true , address=77.90.31.11:4553 , piece_id=OXABXPZRS4F2MOAIOA6F7KH466Q6KV7YXZFPNZBMYO5QW2NM23VQ, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:16.166855959 (632h34m16.037939004s) [noise=true , address=45.91.116.6:4974 , piece_id=W4CUC65I3TLRDC6DILRPEIVNCAIBAIRF54GJ33N4LCA644DLRHEA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: <nil> Jan 1 00:03:16.166880727 (632h34m16.037963762s) [seg_pos={0 0}, upload=554293]: Segment reached optimal threshold of 13 pieces. Jan 1 00:03:16.167120633 (632h34m16.038203698s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54344->145.79.227.17:4970: use of closed network connection Jan 1 00:03:16.167130663 (632h34m16.038213698s) [noise=true , address=145.79.227.17:4970 , piece_id=GJ53Z3ZRHEALAVIEHBC3WVZ4Q66P6LCWEGNCSVPFVNNRAQSCCXYA, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167319702 (632h34m16.038402767s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 1 00:03:16.167363646 (632h34m16.038446701s) [noise=true , address=145.79.227.49:4338 , piece_id=DZEIV4BODEEU5L3IBLMU24NZQPMOCQSQHS52VHP5EHGG7EEJVN3A, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167482202 (632h34m16.038565237s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 1 00:03:16.167491610 (632h34m16.038574645s) [noise=true , address=145.79.227.33:4871 , piece_id=TSUDPE5MOJUQGIUAQY76XAN2BY3HZAXOBYI63EPWGK5654Y6HFFA, seg_pos={0 0}, upload=554293]: Long tail cancellation due to optimal threshold Jan 1 00:03:16.167579778 (632h34m16.038662813s) [seg_pos={0 0}, upload=554293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:03:16.167747627 (632h34m16.038830672s) [upload=554293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:03:16.167755753 (632h34m16.038838798s) [upload=554293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:03:16.496622667 (632h34m16.367705712s) [upload=554293]: Done uploading segments... Jan 1 00:03:16.496646994 (632h34m16.367730029s) [upload=554293]: Upload finished. err: <nil> Duration: 22.72731493s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)