Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X....X....X.....XXX..X..XX.......XXXX....X.X............XXX] duration: 30.00341602s trace: https://jaeger.dev.storj.io/trace/61261969428d488a FAIL (2026-01-01 00:38:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 00:38:14.706937644 (633h9m14.57802068s) [upload=554813]: Starting upload Jan 1 00:38:14.706988642 (633h9m14.578071677s) [upload=554813]: Done starting upload Jan 1 00:38:14.707130041 (633h9m14.578213076s) [upload=554813]: Uploading segments... Jan 1 00:38:14.707292401 (633h9m14.578375446s) [upload=554813]: Got next segment. Inline: false Jan 1 00:38:14.707356292 (633h9m14.578439327s) [upload=554813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:38:14.707365640 (633h9m14.578448685s) [upload=554813]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:38:14.952936235 (633h9m14.82401927s) [seg_pos={0 0}, upload=554813]: Begin upload segment... Jan 1 00:38:14.952977222 (633h9m14.824060257s) [seg_pos={0 0}, upload=554813]: Done begin upload segment. Jan 1 00:38:14.953004685 (633h9m14.82408772s) [noise=true , address=145.79.227.13:4803 , piece_id=HF6ANMVCAK2OG6TH7IKDC24GLTO4KSPIUJ5XVBYAK6JRBCYQGNDA, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.953099736 (633h9m14.824182771s) [noise=true , address=145.79.227.8:4331 , piece_id=QNELRIXSVX42VFRA4NTFATYPKYE7SKVAH44ULHQ5NZWAXUOB3CKQ, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.953552198 (633h9m14.824635263s) [noise=true , address=45.91.116.7:4011 , piece_id=PJICELDMW6EVOB5CGDZTKM6NI5RQLJZBBBZ7I7WI7PGG7OXQBV3Q, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.954254044 (633h9m14.825337079s) [noise=true , address=145.79.227.16:4912 , piece_id=RBEWNF3C2ITE5KT3K2GUGRMZ3QFGTXXQLHLYO7WK7TS2UEWCXGKA, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.954364935 (633h9m14.82544797s) [noise=true , address=145.79.227.50:4374 , piece_id=SK3BXDVZ3NMC5YPO752LFWCJJY7D6ZTXF3TPJJ4AFJXYRN4Y5UJQ, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.954697779 (633h9m14.825780814s) [noise=true , address=145.79.227.38:4979 , piece_id=HDXZ5ELSTF43NBIJHY7ZFLV6EIEX2ABRC2IFLNTPLCQ23DE7A2EQ, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.954809562 (633h9m14.825892597s) [noise=true , address=145.79.227.47:4266 , piece_id=OQEMJ3Y7CB7K56QTL7UQ3KYNZ4VN47PM6MS6CK7IZAGSQWIHYA3A, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.955060700 (633h9m14.826143745s) [noise=true , address=45.91.116.6:4976 , piece_id=WL5HJSLVJKUPJHZCGM7ODEDVZ3K34PMWHLLSB4BGCGYGMFAJIQ3Q, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.955168355 (633h9m14.8262514s) [noise=true , address=145.79.227.37:4566 , piece_id=CJTKWD6CVLSVDHHIM5O5G22K5GTMUHV4NL25IQNJUDXV6BDQ3VSQ, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.955423291 (633h9m14.826506326s) [noise=true , address=45.91.116.13:4156 , piece_id=QRVOZX43P6BNB3LZ7YTSTG6QA65O3KN63OFJCWQWWM3AKVQVK7NQ, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.955505687 (633h9m14.826588722s) [noise=true , address=77.90.31.11:4555 , piece_id=DQS56WIOS2PSYQTSRZBPNCXHHK2FEYSLVRNOI4DB3XYN4AUZGF5A, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.955806571 (633h9m14.826889616s) [noise=true , address=145.79.227.40:4049 , piece_id=5DWXDUZBCMOXKELKP276HOSZZMT55JWSAXG2ZKRHY5XR4OKN542A, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.955918884 (633h9m14.827001919s) [noise=true , address=145.79.227.36:4518 , piece_id=R64ALHDRIZ7FTWMEMFRQGMIRYFLC3V5HIKSWKI5SNBOZI5UEGJOA, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.956293829 (633h9m14.827376884s) [noise=true , address=145.79.227.3:4151 , piece_id=EWTTZU6U4UYWKX3KXEO3E2SLO55PG7X5UJGE4CLT6345WZD4CJZQ, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.956774483 (633h9m14.827857528s) [noise=true , address=145.79.227.25:4619 , piece_id=4MMFZ2SF4MKQKMXW4QAEOZ6L44IR4VM4LX3LMSL2QX6CUX24VKBA, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:14.958157677 (633h9m14.829240722s) [noise=true , address=77.90.31.10:4520 , piece_id=A2WGDZI2TEUUIXU5QMUNRNS3NSP75FYO33JDTPZRFPQTHEZPVSDQ, seg_pos={0 0}, upload=554813]: Uploading piece... Jan 1 00:38:16.658156556 (633h9m16.529239611s) [upload=554813]: Next returned nil segment Jan 1 00:38:16.658174401 (633h9m16.529257456s) [upload=554813]: Waiting for error group managing segments... Jan 1 00:38:16.941979327 (633h9m16.813062362s) [noise=true , address=145.79.227.50:4374 , piece_id=SK3BXDVZ3NMC5YPO752LFWCJJY7D6ZTXF3TPJJ4AFJXYRN4Y5UJQ, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:16.955903831 (633h9m16.826986866s) [noise=true , address=145.79.227.38:4979 , piece_id=HDXZ5ELSTF43NBIJHY7ZFLV6EIEX2ABRC2IFLNTPLCQ23DE7A2EQ, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:16.963876221 (633h9m16.834959266s) [noise=true , address=145.79.227.8:4331 , piece_id=QNELRIXSVX42VFRA4NTFATYPKYE7SKVAH44ULHQ5NZWAXUOB3CKQ, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:16.964144793 (633h9m16.835227828s) [noise=true , address=145.79.227.3:4151 , piece_id=EWTTZU6U4UYWKX3KXEO3E2SLO55PG7X5UJGE4CLT6345WZD4CJZQ, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:16.969922745 (633h9m16.84100578s) [noise=true , address=145.79.227.25:4619 , piece_id=4MMFZ2SF4MKQKMXW4QAEOZ6L44IR4VM4LX3LMSL2QX6CUX24VKBA, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:16.972502256 (633h9m16.843585301s) [noise=true , address=145.79.227.36:4518 , piece_id=R64ALHDRIZ7FTWMEMFRQGMIRYFLC3V5HIKSWKI5SNBOZI5UEGJOA, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:17.025579110 (633h9m16.896662155s) [noise=true , address=145.79.227.37:4566 , piece_id=CJTKWD6CVLSVDHHIM5O5G22K5GTMUHV4NL25IQNJUDXV6BDQ3VSQ, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:17.027602343 (633h9m16.898685388s) [noise=true , address=145.79.227.13:4803 , piece_id=HF6ANMVCAK2OG6TH7IKDC24GLTO4KSPIUJ5XVBYAK6JRBCYQGNDA, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:17.039523492 (633h9m16.910606537s) [noise=true , address=145.79.227.47:4266 , piece_id=OQEMJ3Y7CB7K56QTL7UQ3KYNZ4VN47PM6MS6CK7IZAGSQWIHYA3A, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:17.063693875 (633h9m16.934776911s) [noise=true , address=145.79.227.16:4912 , piece_id=RBEWNF3C2ITE5KT3K2GUGRMZ3QFGTXXQLHLYO7WK7TS2UEWCXGKA, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:17.133432906 (633h9m17.004515951s) [noise=true , address=77.90.31.10:4520 , piece_id=A2WGDZI2TEUUIXU5QMUNRNS3NSP75FYO33JDTPZRFPQTHEZPVSDQ, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:17.301567302 (633h9m17.172650337s) [noise=true , address=45.91.116.13:4156 , piece_id=QRVOZX43P6BNB3LZ7YTSTG6QA65O3KN63OFJCWQWWM3AKVQVK7NQ, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:17.556125320 (633h9m17.427208355s) [noise=true , address=45.91.116.6:4976 , piece_id=WL5HJSLVJKUPJHZCGM7ODEDVZ3K34PMWHLLSB4BGCGYGMFAJIQ3Q, seg_pos={0 0}, upload=554813]: Done uploading piece. err: <nil> Jan 1 00:38:17.556143685 (633h9m17.42722673s) [seg_pos={0 0}, upload=554813]: Segment reached optimal threshold of 13 pieces. Jan 1 00:38:17.556307256 (633h9m17.427390291s) [noise=true , address=45.91.116.7:4011 , piece_id=PJICELDMW6EVOB5CGDZTKM6NI5RQLJZBBBZ7I7WI7PGG7OXQBV3Q, seg_pos={0 0}, upload=554813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Jan 1 00:38:17.556313027 (633h9m17.427396072s) [noise=true , address=45.91.116.7:4011 , piece_id=PJICELDMW6EVOB5CGDZTKM6NI5RQLJZBBBZ7I7WI7PGG7OXQBV3Q, seg_pos={0 0}, upload=554813]: Long tail cancellation due to optimal threshold Jan 1 00:38:17.556483422 (633h9m17.427566457s) [noise=true , address=77.90.31.11:4555 , piece_id=DQS56WIOS2PSYQTSRZBPNCXHHK2FEYSLVRNOI4DB3XYN4AUZGF5A, seg_pos={0 0}, upload=554813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:55512->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:55512->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55512->77.90.31.11:4555: use of closed network connection Jan 1 00:38:17.556494192 (633h9m17.427577247s) [noise=true , address=77.90.31.11:4555 , piece_id=DQS56WIOS2PSYQTSRZBPNCXHHK2FEYSLVRNOI4DB3XYN4AUZGF5A, seg_pos={0 0}, upload=554813]: Long tail cancellation due to optimal threshold Jan 1 00:38:17.556603380 (633h9m17.427686415s) [noise=true , address=145.79.227.40:4049 , piece_id=5DWXDUZBCMOXKELKP276HOSZZMT55JWSAXG2ZKRHY5XR4OKN542A, seg_pos={0 0}, upload=554813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Jan 1 00:38:17.556610694 (633h9m17.427693729s) [noise=true , address=145.79.227.40:4049 , piece_id=5DWXDUZBCMOXKELKP276HOSZZMT55JWSAXG2ZKRHY5XR4OKN542A, seg_pos={0 0}, upload=554813]: Long tail cancellation due to optimal threshold Jan 1 00:38:17.556629660 (633h9m17.427712705s) [seg_pos={0 0}, upload=554813]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:38:17.556694183 (633h9m17.427777218s) [upload=554813]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:38:17.556699333 (633h9m17.427782368s) [upload=554813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:38:17.849728885 (633h9m17.72081193s) [upload=554813]: Done uploading segments... Jan 1 00:38:17.849747911 (633h9m17.720830946s) [upload=554813]: Upload finished. err: <nil> Duration: 30.003442479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...........X..X..............................] duration: 23.036126403s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:38:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 23.036146583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X....X....X.....XXX..X..XX.......XXXX....X.X............XX] duration: 30.002559793s trace: https://jaeger.dev.storj.io/trace/58713abd4e3ab3a7 FAIL (2026-01-01 00:37:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 00:37:14.555108284 (633h8m14.426191329s) [upload=554798]: Starting upload Jan 1 00:37:14.555185140 (633h8m14.426268185s) [upload=554798]: Done starting upload Jan 1 00:37:14.555258209 (633h8m14.426341254s) [upload=554798]: Uploading segments... Jan 1 00:37:14.555391503 (633h8m14.426474538s) [upload=554798]: Got next segment. Inline: false Jan 1 00:37:14.555487525 (633h8m14.42657057s) [upload=554798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:37:14.555495461 (633h8m14.426578496s) [upload=554798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:37:14.799750140 (633h8m14.670833175s) [seg_pos={0 0}, upload=554798]: Begin upload segment... Jan 1 00:37:14.799810094 (633h8m14.670893129s) [seg_pos={0 0}, upload=554798]: Done begin upload segment. Jan 1 00:37:14.799845892 (633h8m14.670928927s) [noise=true , address=145.79.227.5:4221 , piece_id=DEI2FUBFVTQ2GRAAW4RNX5RGDE4HTUPWRVJVM4MJ476P7V7PY7LQ, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.799932166 (633h8m14.671015211s) [noise=true , address=45.91.116.6:4973 , piece_id=WYTZOSCIA3A6RWJYOJFW7PJGUBT2JOIHRPV43PWAJKFB6FHQOUCA, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.800495599 (633h8m14.671578644s) [noise=true , address=45.91.116.12:4227 , piece_id=RCZGZXMJZ4XV6776YV5XUN2K73K7AAA4TBK6G3OUFNHMR6JZUIMA, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.800894529 (633h8m14.671977564s) [noise=true , address=145.79.227.26:4654 , piece_id=FSQL325P52ZD5APAGU3FO4CQRJRNNUVQQ33OFE6U55Z4KPP4HCEQ, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.801178189 (633h8m14.672261224s) [noise=true , address=145.79.227.38:4978 , piece_id=455Y32RG3TTAXGWTJTRETXSYRQXHYWECVQOCI2F43TP4BQ4PEXRQ, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.801294720 (633h8m14.672377765s) [noise=true , address=145.79.227.37:4566 , piece_id=RWEYQDEDZ6FIKQXBQJJVZ6YNCAXH7Q4PYYW433WXP2N6FZVY3LHA, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.801459585 (633h8m14.67254262s) [noise=true , address=145.79.227.11:4730 , piece_id=PRJNHTCR6KEOKXMRCQGQR46ZPVJ46GS22HF3PCLEJQLAOOV3BZCA, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.801719459 (633h8m14.672802494s) [noise=true , address=145.79.227.52:4445 , piece_id=2XC2QSRMMVRPOGRIABKZDRTPYVBVHLQCUR2EPWJMCTWRUUVSBIBQ, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.802021013 (633h8m14.673104049s) [noise=true , address=45.91.116.11:4120 , piece_id=TUN5LX44IKG7IHNVZEX6ASY7PIB7F55AJXSEJYD6FWHXSPT7IOHQ, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.802063995 (633h8m14.67314703s) [noise=true , address=145.79.227.3:4151 , piece_id=CLQGTE7UPTSQ2TAAQY766ZWGYBLSHRXPVGNFGLPQYLULR3CG3HYA, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.802350510 (633h8m14.673433555s) [noise=true , address=77.90.31.17:4770 , piece_id=22VZZGOQOMW2TO36VXATNLZQGOALFN2GLGZXUWYNZ5HODV632FEQ, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.802408401 (633h8m14.673491436s) [noise=true , address=145.79.227.15:4873 , piece_id=ZKE7LFDCEJ2QXX5ZGTGVQHB4O7WYDG4UPEGAMLRUMJM4M2TOPYHA, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.802772915 (633h8m14.67385596s) [noise=true , address=145.79.227.23:4547 , piece_id=UJVUXSRPIOSA7YRYUFL2YW32MGQZO5ADUIZ7Z33KA4QSJYYXSQVQ, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.803095369 (633h8m14.674178414s) [noise=true , address=145.79.227.40:4051 , piece_id=O6AD3XW3VHAT2E2WZTE4YVC33W4Q7SSO4SQYPQFHME5L7UHVEDBQ, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.803654945 (633h8m14.67473799s) [noise=true , address=145.79.227.46:4602 , piece_id=N2R6VVAVEDLJ5NTV2L6PIXGJ74S5QT7OZRM36DTL76WWHF6SS6PQ, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:14.804252553 (633h8m14.675335588s) [noise=true , address=45.91.116.5:4298 , piece_id=N2LOLNUASYLKQYO33SFBC3VF3N5ICCFR5JI6WDMZEP2FEI5JBOFA, seg_pos={0 0}, upload=554798]: Uploading piece... Jan 1 00:37:16.500788830 (633h8m16.371871875s) [upload=554798]: Next returned nil segment Jan 1 00:37:16.500812735 (633h8m16.37189577s) [upload=554798]: Waiting for error group managing segments... Jan 1 00:37:16.783844059 (633h8m16.654927094s) [noise=true , address=145.79.227.26:4654 , piece_id=FSQL325P52ZD5APAGU3FO4CQRJRNNUVQQ33OFE6U55Z4KPP4HCEQ, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:16.787279339 (633h8m16.658362384s) [noise=true , address=145.79.227.11:4730 , piece_id=PRJNHTCR6KEOKXMRCQGQR46ZPVJ46GS22HF3PCLEJQLAOOV3BZCA, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:16.787520849 (633h8m16.658603894s) [noise=true , address=145.79.227.46:4602 , piece_id=N2R6VVAVEDLJ5NTV2L6PIXGJ74S5QT7OZRM36DTL76WWHF6SS6PQ, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:16.790355857 (633h8m16.661438892s) [noise=true , address=145.79.227.5:4221 , piece_id=DEI2FUBFVTQ2GRAAW4RNX5RGDE4HTUPWRVJVM4MJ476P7V7PY7LQ, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:16.794959945 (633h8m16.666043s) [noise=true , address=145.79.227.15:4873 , piece_id=ZKE7LFDCEJ2QXX5ZGTGVQHB4O7WYDG4UPEGAMLRUMJM4M2TOPYHA, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:16.796899056 (633h8m16.667982092s) [noise=true , address=145.79.227.23:4547 , piece_id=UJVUXSRPIOSA7YRYUFL2YW32MGQZO5ADUIZ7Z33KA4QSJYYXSQVQ, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:16.804307363 (633h8m16.675390398s) [noise=true , address=145.79.227.38:4978 , piece_id=455Y32RG3TTAXGWTJTRETXSYRQXHYWECVQOCI2F43TP4BQ4PEXRQ, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:16.819863604 (633h8m16.690946639s) [noise=true , address=145.79.227.3:4151 , piece_id=CLQGTE7UPTSQ2TAAQY766ZWGYBLSHRXPVGNFGLPQYLULR3CG3HYA, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:16.881923863 (633h8m16.753006898s) [noise=true , address=145.79.227.52:4445 , piece_id=2XC2QSRMMVRPOGRIABKZDRTPYVBVHLQCUR2EPWJMCTWRUUVSBIBQ, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:16.953146078 (633h8m16.824229113s) [noise=true , address=145.79.227.37:4566 , piece_id=RWEYQDEDZ6FIKQXBQJJVZ6YNCAXH7Q4PYYW433WXP2N6FZVY3LHA, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:16.981578647 (633h8m16.852661692s) [noise=true , address=77.90.31.17:4770 , piece_id=22VZZGOQOMW2TO36VXATNLZQGOALFN2GLGZXUWYNZ5HODV632FEQ, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:17.145740320 (633h8m17.016823355s) [noise=true , address=45.91.116.11:4120 , piece_id=TUN5LX44IKG7IHNVZEX6ASY7PIB7F55AJXSEJYD6FWHXSPT7IOHQ, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:17.159716031 (633h8m17.030799066s) [noise=true , address=45.91.116.5:4298 , piece_id=N2LOLNUASYLKQYO33SFBC3VF3N5ICCFR5JI6WDMZEP2FEI5JBOFA, seg_pos={0 0}, upload=554798]: Done uploading piece. err: <nil> Jan 1 00:37:17.159741500 (633h8m17.030824545s) [seg_pos={0 0}, upload=554798]: Segment reached optimal threshold of 13 pieces. Jan 1 00:37:17.160015251 (633h8m17.031098316s) [noise=true , address=45.91.116.6:4973 , piece_id=WYTZOSCIA3A6RWJYOJFW7PJGUBT2JOIHRPV43PWAJKFB6FHQOUCA, seg_pos={0 0}, upload=554798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Jan 1 00:37:17.160025190 (633h8m17.031108225s) [noise=true , address=45.91.116.6:4973 , piece_id=WYTZOSCIA3A6RWJYOJFW7PJGUBT2JOIHRPV43PWAJKFB6FHQOUCA, seg_pos={0 0}, upload=554798]: Long tail cancellation due to optimal threshold Jan 1 00:37:17.160221714 (633h8m17.03130477s) [noise=true , address=45.91.116.12:4227 , piece_id=RCZGZXMJZ4XV6776YV5XUN2K73K7AAA4TBK6G3OUFNHMR6JZUIMA, seg_pos={0 0}, upload=554798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jan 1 00:37:17.160231333 (633h8m17.031314398s) [noise=true , address=45.91.116.12:4227 , piece_id=RCZGZXMJZ4XV6776YV5XUN2K73K7AAA4TBK6G3OUFNHMR6JZUIMA, seg_pos={0 0}, upload=554798]: Long tail cancellation due to optimal threshold Jan 1 00:37:17.160252704 (633h8m17.031335749s) [noise=true , address=145.79.227.40:4051 , piece_id=O6AD3XW3VHAT2E2WZTE4YVC33W4Q7SSO4SQYPQFHME5L7UHVEDBQ, seg_pos={0 0}, upload=554798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Jan 1 00:37:17.160272200 (633h8m17.031355235s) [noise=true , address=145.79.227.40:4051 , piece_id=O6AD3XW3VHAT2E2WZTE4YVC33W4Q7SSO4SQYPQFHME5L7UHVEDBQ, seg_pos={0 0}, upload=554798]: Long tail cancellation due to optimal threshold Jan 1 00:37:17.160332205 (633h8m17.03141524s) [seg_pos={0 0}, upload=554798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:37:17.160519781 (633h8m17.031602827s) [upload=554798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:37:17.160528038 (633h8m17.031611083s) [upload=554798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:37:17.467174879 (633h8m17.338257924s) [upload=554798]: Done uploading segments... Jan 1 00:37:17.467192782 (633h8m17.338275827s) [upload=554798]: Upload finished. err: <nil> Duration: 30.002570323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X....X....X.....XXX..X..XX.......XXXX....X.X............X] duration: 30.002897716s trace: https://jaeger.dev.storj.io/trace/23125a6bdb4b5f3 FAIL (2026-01-01 00:37:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: PUQF44WAQ6X5474YTR5RL2DF7M47OKNUWFFYCRR6Y3Q3UAZUVMVQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 4P5YLJNUV6JYGCDODISK76ARXM4XWW2OX2WDCNB4XSQEYO5MWTFQ) context deadline exceeded ======LOGS====== Jan 1 00:36:44.337408130 (633h7m44.208491175s) [upload=554788]: Starting upload Jan 1 00:36:44.337507269 (633h7m44.208590314s) [upload=554788]: Done starting upload Jan 1 00:36:44.337569167 (633h7m44.208652212s) [upload=554788]: Uploading segments... Jan 1 00:36:44.337729211 (633h7m44.208812246s) [upload=554788]: Got next segment. Inline: false Jan 1 00:36:44.337786450 (633h7m44.208869485s) [upload=554788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:36:44.337792952 (633h7m44.208875997s) [upload=554788]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:36:44.583178024 (633h7m44.454261079s) [seg_pos={0 0}, upload=554788]: Begin upload segment... Jan 1 00:36:44.583250662 (633h7m44.454333707s) [seg_pos={0 0}, upload=554788]: Done begin upload segment. Jan 1 00:36:44.583308813 (633h7m44.454391858s) [noise=true , address=145.79.227.15:4875 , piece_id=4VGQ3MUAKZT3GLAA25NEA7C3ASLEWJJ7BRTUCJHF7YC6WCYTXUNQ, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.583355241 (633h7m44.454438286s) [noise=true , address=145.79.227.42:4122 , piece_id=LNEXS2R6POIJDN6JDUVDJ37KCFLSJ2DNVDCCJYZXNETEJZOA4QCQ, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.584008155 (633h7m44.45509119s) [noise=true , address=145.79.227.45:4231 , piece_id=77W4XHQZPDZQ5LKFZM6QDDWRAP677ZTWHBLO6EAR2HVKTSE7MO6A, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.584485293 (633h7m44.455568328s) [noise=true , address=145.79.227.6:4258 , piece_id=YXQ4DO3CEV7JSSRQC5MWIOR4IYNGLIKM6KB36FDEAW4P4AHWGSTA, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.585626205 (633h7m44.45670924s) [noise=true , address=45.91.116.14:4192 , piece_id=3ZJMWNWU6AO5PX5JYNQMEEH4IVZVLRZQIGVBQ4LDFJQDINUM2P7Q, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.587697689 (633h7m44.458780734s) [noise=true , address=45.91.116.8:4048 , piece_id=53ZCB5DTDBJZGOYL6MVK6GLEKTFK4U65PFYTIHXU7MPRKZ4ADQBQ, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.587810815 (633h7m44.45889386s) [noise=true , address=145.79.227.18:4006 , piece_id=FCKULQTRYWAR2ZQGMSFGN2PZRZJAT7OV3TTQ7QYDTUVP73O4RZMQ, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.590168244 (633h7m44.461251289s) [noise=true , address=77.90.31.11:4556 , piece_id=7NQFJ6B7MHBZ373APSCNJIBL23G56MDHBTGYICTPJZCGIORZ3H6A, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.593115456 (633h7m44.464198491s) [noise=true , address=145.79.227.48:4302 , piece_id=M4QNWSNMNNDIOSMQLFC3Z44F6K3NJFWQBZWJ3YBA4KYNQ77CRI6Q, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.596188808 (633h7m44.467271843s) [noise=true , address=45.91.116.13:4153 , piece_id=PUQF44WAQ6X5474YTR5RL2DF7M47OKNUWFFYCRR6Y3Q3UAZUVMVQ, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.598935919 (633h7m44.470018954s) [noise=true , address=145.79.227.16:4909 , piece_id=SSO6RIHSBMS7X3PMAOWTWCBZPF4QSLC2SN6XIKOQQ4XA45NURCHQ, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.600196449 (633h7m44.471279484s) [noise=true , address=145.79.227.39:4013 , piece_id=BGCHE2NC5ZZIUFMDZJWMFZXK3FOK7SAGERQ6H7AA5WFK2622XVXA, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.601211692 (633h7m44.472294727s) [noise=true , address=45.91.116.11:4117 , piece_id=WH4TK2IEGJ4N77YUZAY6BKOCUMQQL5QFFNVONF3LRFKXNTA3OW5Q, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.602233277 (633h7m44.473316322s) [noise=true , address=45.91.116.7:4010 , piece_id=4P5YLJNUV6JYGCDODISK76ARXM4XWW2OX2WDCNB4XSQEYO5MWTFQ, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.602518500 (633h7m44.473601535s) [noise=true , address=145.79.227.19:4078 , piece_id=FREUBTVF4M5A5WZT7AST5VT2V4Q7Z34AISRHVSRBCMBFETYQGPBQ, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:44.604073822 (633h7m44.475156867s) [noise=true , address=145.79.227.13:4804 , piece_id=NZX7C6MFJPLWNUZZ6PWQKFLCW4AVG2KXSADR4QXNVHSFESTJTTSQ, seg_pos={0 0}, upload=554788]: Uploading piece... Jan 1 00:36:46.274047410 (633h7m46.145130445s) [upload=554788]: Next returned nil segment Jan 1 00:36:46.274062529 (633h7m46.145145564s) [upload=554788]: Waiting for error group managing segments... Jan 1 00:36:46.564399404 (633h7m46.435482439s) [noise=true , address=145.79.227.45:4231 , piece_id=77W4XHQZPDZQ5LKFZM6QDDWRAP677ZTWHBLO6EAR2HVKTSE7MO6A, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:46.595141730 (633h7m46.466224765s) [noise=true , address=145.79.227.48:4302 , piece_id=M4QNWSNMNNDIOSMQLFC3Z44F6K3NJFWQBZWJ3YBA4KYNQ77CRI6Q, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:46.611206840 (633h7m46.482289885s) [noise=true , address=145.79.227.15:4875 , piece_id=4VGQ3MUAKZT3GLAA25NEA7C3ASLEWJJ7BRTUCJHF7YC6WCYTXUNQ, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:46.622147412 (633h7m46.493230447s) [noise=true , address=145.79.227.13:4804 , piece_id=NZX7C6MFJPLWNUZZ6PWQKFLCW4AVG2KXSADR4QXNVHSFESTJTTSQ, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:46.632061119 (633h7m46.503144154s) [noise=true , address=145.79.227.39:4013 , piece_id=BGCHE2NC5ZZIUFMDZJWMFZXK3FOK7SAGERQ6H7AA5WFK2622XVXA, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:46.632376279 (633h7m46.503459315s) [noise=true , address=145.79.227.18:4006 , piece_id=FCKULQTRYWAR2ZQGMSFGN2PZRZJAT7OV3TTQ7QYDTUVP73O4RZMQ, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:46.671399814 (633h7m46.542482849s) [noise=true , address=145.79.227.16:4909 , piece_id=SSO6RIHSBMS7X3PMAOWTWCBZPF4QSLC2SN6XIKOQQ4XA45NURCHQ, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:46.754166401 (633h7m46.625249446s) [noise=true , address=77.90.31.11:4556 , piece_id=7NQFJ6B7MHBZ373APSCNJIBL23G56MDHBTGYICTPJZCGIORZ3H6A, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:46.896426761 (633h7m46.767509796s) [noise=true , address=45.91.116.8:4048 , piece_id=53ZCB5DTDBJZGOYL6MVK6GLEKTFK4U65PFYTIHXU7MPRKZ4ADQBQ, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:46.957568191 (633h7m46.828651226s) [noise=true , address=145.79.227.6:4258 , piece_id=YXQ4DO3CEV7JSSRQC5MWIOR4IYNGLIKM6KB36FDEAW4P4AHWGSTA, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:46.977859909 (633h7m46.848942954s) [noise=true , address=45.91.116.7:4010 , piece_id=4P5YLJNUV6JYGCDODISK76ARXM4XWW2OX2WDCNB4XSQEYO5MWTFQ, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:47.667992159 (633h7m47.539075194s) [noise=true , address=45.91.116.11:4117 , piece_id=WH4TK2IEGJ4N77YUZAY6BKOCUMQQL5QFFNVONF3LRFKXNTA3OW5Q, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:48.026900490 (633h7m47.897983525s) [noise=true , address=45.91.116.13:4153 , piece_id=PUQF44WAQ6X5474YTR5RL2DF7M47OKNUWFFYCRR6Y3Q3UAZUVMVQ, seg_pos={0 0}, upload=554788]: Done uploading piece. err: <nil> Jan 1 00:36:48.026921110 (633h7m47.898004145s) [seg_pos={0 0}, upload=554788]: Segment reached optimal threshold of 13 pieces. Jan 1 00:36:48.027172298 (633h7m47.898255333s) [noise=true , address=45.91.116.14:4192 , piece_id=3ZJMWNWU6AO5PX5JYNQMEEH4IVZVLRZQIGVBQ4LDFJQDINUM2P7Q, seg_pos={0 0}, upload=554788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 1 00:36:48.027183239 (633h7m47.898266284s) [noise=true , address=45.91.116.14:4192 , piece_id=3ZJMWNWU6AO5PX5JYNQMEEH4IVZVLRZQIGVBQ4LDFJQDINUM2P7Q, seg_pos={0 0}, upload=554788]: Long tail cancellation due to optimal threshold Jan 1 00:36:48.027456429 (633h7m47.898539464s) [noise=true , address=145.79.227.42:4122 , piece_id=LNEXS2R6POIJDN6JDUVDJ37KCFLSJ2DNVDCCJYZXNETEJZOA4QCQ, seg_pos={0 0}, upload=554788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Jan 1 00:36:48.027475305 (633h7m47.89855834s) [noise=true , address=145.79.227.42:4122 , piece_id=LNEXS2R6POIJDN6JDUVDJ37KCFLSJ2DNVDCCJYZXNETEJZOA4QCQ, seg_pos={0 0}, upload=554788]: Long tail cancellation due to optimal threshold Jan 1 00:36:48.027531872 (633h7m47.898614908s) [noise=true , address=145.79.227.19:4078 , piece_id=FREUBTVF4M5A5WZT7AST5VT2V4Q7Z34AISRHVSRBCMBFETYQGPBQ, seg_pos={0 0}, upload=554788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:38788->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:38788->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38788->145.79.227.19:4078: use of closed network connection Jan 1 00:36:48.027540129 (633h7m47.898623164s) [noise=true , address=145.79.227.19:4078 , piece_id=FREUBTVF4M5A5WZT7AST5VT2V4Q7Z34AISRHVSRBCMBFETYQGPBQ, seg_pos={0 0}, upload=554788]: Long tail cancellation due to optimal threshold Jan 1 00:36:48.027558433 (633h7m47.898641468s) [seg_pos={0 0}, upload=554788]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:36:48.027630270 (633h7m47.898713305s) [upload=554788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:36:48.027637454 (633h7m47.89872049s) [upload=554788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:36:48.324602870 (633h7m48.195685905s) [upload=554788]: Done uploading segments... Jan 1 00:36:48.324621796 (633h7m48.195704841s) [upload=554788]: Upload finished. err: <nil> Duration: 30.002919998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X...........X..X.............................] duration: 15.417315206s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:37:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 15.417340153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X....X....X.....XXX..X..XX.......XXXX....X.X............] duration: 5.89448602s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:36:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 5.894516077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X...........X..X............................] duration: 6.848579518s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:36:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 6.848610246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X...........X..X...........................] duration: 20.710156931s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:35:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 20.710183533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X....X....X.....XXX..X..XX.......XXXX....X.X...........] duration: 6.071147132s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:35:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 6.07117697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X....X....X.....XXX..X..XX.......XXXX....X.X..........] duration: 6.343657215s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:35:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 6.343688795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X...........X..X..........................] duration: 21.751984728s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:35:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 21.752010166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X......X....X....X.....XXX..X..XX.......XXXX....X.X.........] duration: 23.505963223s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:34:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 23.505978682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X...........X..X.........................] duration: 27.175015697s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:34:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 27.17504432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X......X....X....X.....XXX..X..XX.......XXXX....X.X........] duration: 6.24257512s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:33:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 6.242597774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X......X....X....X.....XXX..X..XX.......XXXX....X.X.......] duration: 6.468527513s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:33:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 6.468556177s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X...........X..X........................] duration: 16.774919885s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:33:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 16.774945482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....................X...........X..X.......................] duration: 15.134800661s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:32:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 15.134832543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X......X....X....X.....XXX..X..XX.......XXXX....X.X......] duration: 6.131936219s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:32:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 6.131967878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....................X...........X..X......................] duration: 28.924208692s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:31:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 28.924241323s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X......X....X....X.....XXX..X..XX.......XXXX....X.X.....] duration: 5.966676187s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:31:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 5.966695894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X....X......X....X....X.....XXX..X..XX.......XXXX....X.X....] duration: 9.509362162s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:31:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 9.509388753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X....................X...........X..X.....................] duration: 28.079787992s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:31:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 28.079808683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X....X......X....X....X.....XXX..X..XX.......XXXX....X.X...] duration: 5.861910414s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:30:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 5.862009472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X....................X...........X..X....................] duration: 45.786865108s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:30:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 45.786895897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X....X......X....X....X.....XXX..X..XX.......XXXX....X.X..] duration: 6.169062962s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:30:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 6.169085295s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X....................X...........X..X...................] duration: 38.688213297s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:29:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 38.688236519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X....X......X....X....X.....XXX..X..XX.......XXXX....X.X.] duration: 6.731778072s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:29:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 6.73190775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X....X......X....X....X.....XXX..X..XX.......XXXX....X.X] duration: 30.002065444s trace: https://jaeger.dev.storj.io/trace/1c14f06d73e414e4 FAIL (2026-01-01 00:29:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ) context deadline exceeded ======LOGS====== 73]: Done starting upload Jan 1 00:29:01.476113228 (633h0m1.347196273s) [upload=554673]: Uploading segments... Jan 1 00:29:01.476295705 (633h0m1.34737875s) [upload=554673]: Got next segment. Inline: false Jan 1 00:29:01.476350359 (633h0m1.347433404s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:29:01.476358976 (633h0m1.347442021s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:29:01.729373474 (633h0m1.600456519s) [seg_pos={0 0}, upload=554673]: Begin upload segment... Jan 1 00:29:01.729437406 (633h0m1.600520451s) [seg_pos={0 0}, upload=554673]: Done begin upload segment. Jan 1 00:29:01.729472863 (633h0m1.600555919s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729489545 (633h0m1.60057259s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.729564568 (633h0m1.600647614s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730111890 (633h0m1.601194945s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.730742671 (633h0m1.601825726s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.731174283 (633h0m1.602257328s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.733635329 (633h0m1.604718374s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.735696854 (633h0m1.606779899s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.737711901 (633h0m1.608794946s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.739564849 (633h0m1.610647905s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.741248415 (633h0m1.61233145s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.743326292 (633h0m1.614409337s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.744087902 (633h0m1.615170947s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.745755948 (633h0m1.616838993s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748090944 (633h0m1.619173989s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:01.748360938 (633h0m1.619443983s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Uploading piece... Jan 1 00:29:03.442353041 (633h0m3.313436086s) [upload=554673]: Next returned nil segment Jan 1 00:29:03.442371286 (633h0m3.313454321s) [upload=554673]: Waiting for error group managing segments... Jan 1 00:29:03.727769888 (633h0m3.598852933s) [noise=true , address=145.79.227.15:4874 , piece_id=5S5DQKAEV3FGUQ7O2XEIIR2QZP2RITFOBDBXFHBFJKGGIBID4ZDA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.742632888 (633h0m3.613715933s) [noise=true , address=145.79.227.2:4115 , piece_id=YE3A7DFDQHV7VEQ7637I36IQYXTXEZV4TH7DP7DVHTRDIRXPCHHA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.743085861 (633h0m3.614168896s) [noise=true , address=145.79.227.48:4302 , piece_id=OUVMMSXXIIHFTNR4DUNTB6QODSPWWRXREXR23XJ2CA7OK3UECFZQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.798770751 (633h0m3.669853796s) [noise=true , address=145.79.227.3:4149 , piece_id=XCQ5KBAB7WM7AV5ZNLUYAFB32NG72PUOEVWRWHNMMEQYQ5ZWTAXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.802163963 (633h0m3.673247008s) [noise=true , address=145.79.227.4:4186 , piece_id=FQZ3VP2IYKS2PNHOLINZVG5FJ7T2WPSABAA57DUZ6CILVGEQKQFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.806677867 (633h0m3.677760912s) [noise=true , address=145.79.227.42:4123 , piece_id=ECJFHAMCUNCBB3EJMWXDMRP4BSZMNAGEMVHKZGSLEF5QWSHLI2WQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.809470364 (633h0m3.680553399s) [noise=true , address=145.79.227.38:4977 , piece_id=4JV3U4XRATKO2OXZBW2XNM2VASF7ALALFVLLVTAOHQ5ZDVG4IFBQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.810039498 (633h0m3.681122533s) [noise=true , address=145.79.227.12:4765 , piece_id=2Q7V4PMQZBEJUP3MPCDON3HP7BIRNC2NKFQSLFUGQRCHBKMSUYFQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:03.904068250 (633h0m3.775151295s) [noise=true , address=77.90.31.19:4844 , piece_id=ZB7OAXVGX5WYBPCHWXBKB35PKYUTLJYB6NUYF22S3DE343PT5AJQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.016962926 (633h0m3.888045971s) [noise=true , address=77.90.31.14:4662 , piece_id=I5HNCJ3BWDMOQHEBP24CMWWYBIMYIZL75CQHHVF4FKDJ34RFXC5Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.110092194 (633h0m3.98117524s) [noise=true , address=45.91.116.6:4974 , piece_id=CRKDES2SSLZZC2DPW3H5K5T6U2ULT7UFGT6ZQ7QQOIMUSDAPYAPA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.139056375 (633h0m4.01013942s) [noise=true , address=45.91.116.13:4156 , piece_id=MT7PD5KPJCG6BDG3CVNRZYEVVCL7RSLU6QDDZP45CSEZLCJS3W6Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158379599 (633h0m4.029462634s) [noise=true , address=45.91.116.9:4262 , piece_id=GS6F4B2SHFNWDGDUVAZ2XR42L6THH672EYMV5LOVPZM3GCYNCISQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: <nil> Jan 1 00:29:04.158399887 (633h0m4.029482932s) [seg_pos={0 0}, upload=554673]: Segment reached optimal threshold of 13 pieces. Jan 1 00:29:04.158717942 (633h0m4.029800977s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43654->77.90.31.12:4590: use of closed network connection Jan 1 00:29:04.158724185 (633h0m4.02980723s) [noise=true , address=77.90.31.12:4590 , piece_id=IQFCVRJUIJPB3MNOP5PYBWUPRQCR5YQSFQDE4PIAE4GGZA4J6R2Q, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158906963 (633h0m4.029990008s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 1 00:29:04.158929034 (633h0m4.030012079s) [noise=true , address=145.79.227.34:4907 , piece_id=UCQKC7QQMVOKA6BG6GOLL7BVJFHWEAARPLSXZGANUFCIRZ6ZD2WA, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.158954483 (633h0m4.030037518s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58938->45.91.116.5:4300: use of closed network connection Jan 1 00:29:04.158963310 (633h0m4.030046345s) [noise=true , address=45.91.116.5:4300 , piece_id=BYWRHBYZR6C3CBAASP3XHY72REXNUCHFHNYJQSBHJNFROHU7SUXQ, seg_pos={0 0}, upload=554673]: Long tail cancellation due to optimal threshold Jan 1 00:29:04.159065624 (633h0m4.03014867s) [seg_pos={0 0}, upload=554673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:29:04.159196865 (633h0m4.03027992s) [upload=554673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:29:04.159204018 (633h0m4.030287063s) [upload=554673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:29:04.448917397 (633h0m4.320000452s) [upload=554673]: Done uploading segments... Jan 1 00:29:04.448942454 (633h0m4.320025499s) [upload=554673]: Upload finished. err: <nil> Duration: 30.002087426s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X....................X...........X..X..................] duration: 17.273672798s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:29:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 17.273698606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X....X....X......X....X....X.....XXX..X..XX.......XXXX....X.] duration: 6.17608205s trace: https://jaeger.dev.storj.io/trace/547ec6510f17298a FAIL (2026-01-01 00:27:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: OKVOAOBKNQKK4EDGZFPOEUZIFQUSMXE75OKRRIXLM3LBUV3KLOZQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: DSUJD6UY2UDUWZOYYT5DG5EYP65NHCYSHINICQEBI45I4XRLKURQ) context deadline exceeded ======LOGS====== Jan 1 00:26:43.639934012 (632h57m43.511017057s) [upload=554639]: Starting upload Jan 1 00:26:43.640013644 (632h57m43.511096689s) [upload=554639]: Done starting upload Jan 1 00:26:43.640065633 (632h57m43.511148668s) [upload=554639]: Uploading segments... Jan 1 00:26:43.640239274 (632h57m43.511322309s) [upload=554639]: Got next segment. Inline: false Jan 1 00:26:43.640275743 (632h57m43.511358778s) [upload=554639]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:26:43.640281293 (632h57m43.511364338s) [upload=554639]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:26:43.887058846 (632h57m43.758141911s) [seg_pos={0 0}, upload=554639]: Begin upload segment... Jan 1 00:26:43.887136624 (632h57m43.758219679s) [seg_pos={0 0}, upload=554639]: Done begin upload segment. Jan 1 00:26:43.887179365 (632h57m43.7582624s) [noise=true , address=145.79.227.52:4445 , piece_id=7NIVFBNHANX2XBLAGD362IZLQODAUXWZ5354YGRDM4QXDAPWGPFQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.887202840 (632h57m43.758285915s) [noise=true , address=145.79.227.15:4876 , piece_id=ZX4FJTXU5MBTN6LDZWP7EZGDPCAQPNWTQGZQ4Q5PKOSRWX42MQ2A, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.888016850 (632h57m43.759099925s) [noise=true , address=77.90.31.17:4770 , piece_id=PJ26PIZDCQEXNZGSEGQUQBNLEOCNKARC4UD3DM7WDZOXBLBU47UA, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.888297224 (632h57m43.759380259s) [noise=true , address=77.90.31.13:4629 , piece_id=IS73FV4O3SAIBO323D7I36BI7C42MNL3ZXTTTFW54YS3Z6HTKO5A, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.888591404 (632h57m43.759674439s) [noise=true , address=145.79.227.11:4732 , piece_id=6X2Z4KXFYZS2JDVEIEYDOTPKDGFUY4UBTHYVZZ6MAYO2UINBEGMQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.888609038 (632h57m43.759692083s) [noise=true , address=45.91.116.10:4081 , piece_id=544HX4WZAGWRHVT5YAM7NOSXSABIR6RTJ35KGDVFLYHLFTGDM4LQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.888838624 (632h57m43.759921659s) [noise=true , address=145.79.227.48:4303 , piece_id=3D2VXRQ4A3WBZSNFROC3A5GFQG53PDZXJZGDPOA7IVE2SYZTKNYA, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889015943 (632h57m43.760098978s) [noise=true , address=45.91.116.5:4300 , piece_id=YPTEYUEENQ4CADNPMAQNQ3VD44MX27VVPJKU7HBLSMQQ2NPDOCTQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889078681 (632h57m43.760161716s) [noise=true , address=45.91.116.12:4228 , piece_id=OKVOAOBKNQKK4EDGZFPOEUZIFQUSMXE75OKRRIXLM3LBUV3KLOZQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889373012 (632h57m43.760456047s) [noise=true , address=145.79.227.14:4839 , piece_id=DMZJWMWXX3T45OBC3XTC5H7VK7Q2YNKLVNCHCPFXFQ5P3W3NAZ4A, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889439960 (632h57m43.760523005s) [noise=true , address=145.79.227.9:4367 , piece_id=MD277W6E5CIWCOARSFHXEF2ADMF6CJUOYFYXGPVJP7F364KWQCFA, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889644059 (632h57m43.760727094s) [noise=true , address=45.91.116.13:4153 , piece_id=DSUJD6UY2UDUWZOYYT5DG5EYP65NHCYSHINICQEBI45I4XRLKURQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889922669 (632h57m43.761005704s) [noise=true , address=145.79.227.46:4601 , piece_id=Y45NHDBQV7ANRWZGYJX2Q53KOEF3TWHWYFCEOYFSUPKQD662USYQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.890240364 (632h57m43.761323409s) [noise=true , address=77.90.31.19:4841 , piece_id=HZ3R6TCUSLAA5RWH4HQN4QRDVNXG2PRDQLKIQIKDHDEESR6YNQVQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.890690932 (632h57m43.761773967s) [noise=true , address=145.79.227.38:4979 , piece_id=BGSLO7XLEC6G3EFOWBZRJW7OJFGLZPHFBGELNBHYMCIM4YIUJP2A, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.890998016 (632h57m43.762081051s) [noise=true , address=145.79.227.39:4014 , piece_id=6INN6YZ75DVM3FY5IZV3BHLATS3IRR6RHOLA3G7ZCANKCYZLOXKA, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:45.587750382 (632h57m45.458833417s) [upload=554639]: Next returned nil segment Jan 1 00:26:45.587861183 (632h57m45.458944238s) [upload=554639]: Waiting for error group managing segments... Jan 1 00:26:45.861879976 (632h57m45.732963011s) [noise=true , address=145.79.227.52:4445 , piece_id=7NIVFBNHANX2XBLAGD362IZLQODAUXWZ5354YGRDM4QXDAPWGPFQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.877011187 (632h57m45.748094222s) [noise=true , address=145.79.227.14:4839 , piece_id=DMZJWMWXX3T45OBC3XTC5H7VK7Q2YNKLVNCHCPFXFQ5P3W3NAZ4A, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.877073907 (632h57m45.748156952s) [noise=true , address=145.79.227.48:4303 , piece_id=3D2VXRQ4A3WBZSNFROC3A5GFQG53PDZXJZGDPOA7IVE2SYZTKNYA, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.888489253 (632h57m45.759572288s) [noise=true , address=145.79.227.46:4601 , piece_id=Y45NHDBQV7ANRWZGYJX2Q53KOEF3TWHWYFCEOYFSUPKQD662USYQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.890341650 (632h57m45.761424695s) [noise=true , address=145.79.227.15:4876 , piece_id=ZX4FJTXU5MBTN6LDZWP7EZGDPCAQPNWTQGZQ4Q5PKOSRWX42MQ2A, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.890663152 (632h57m45.761746187s) [noise=true , address=145.79.227.11:4732 , piece_id=6X2Z4KXFYZS2JDVEIEYDOTPKDGFUY4UBTHYVZZ6MAYO2UINBEGMQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.898816437 (632h57m45.769899472s) [noise=true , address=145.79.227.39:4014 , piece_id=6INN6YZ75DVM3FY5IZV3BHLATS3IRR6RHOLA3G7ZCANKCYZLOXKA, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.899081672 (632h57m45.770164707s) [noise=true , address=145.79.227.9:4367 , piece_id=MD277W6E5CIWCOARSFHXEF2ADMF6CJUOYFYXGPVJP7F364KWQCFA, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.962848973 (632h57m45.833932008s) [noise=true , address=145.79.227.38:4979 , piece_id=BGSLO7XLEC6G3EFOWBZRJW7OJFGLZPHFBGELNBHYMCIM4YIUJP2A, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:46.190330593 (632h57m46.061413628s) [noise=true , address=45.91.116.12:4228 , piece_id=OKVOAOBKNQKK4EDGZFPOEUZIFQUSMXE75OKRRIXLM3LBUV3KLOZQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:46.239117618 (632h57m46.110200654s) [noise=true , address=45.91.116.13:4153 , piece_id=DSUJD6UY2UDUWZOYYT5DG5EYP65NHCYSHINICQEBI45I4XRLKURQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:46.264420027 (632h57m46.135503062s) [noise=true , address=45.91.116.5:4300 , piece_id=YPTEYUEENQ4CADNPMAQNQ3VD44MX27VVPJKU7HBLSMQQ2NPDOCTQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:46.275234450 (632h57m46.146317485s) [noise=true , address=45.91.116.10:4081 , piece_id=544HX4WZAGWRHVT5YAM7NOSXSABIR6RTJ35KGDVFLYHLFTGDM4LQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:46.275253165 (632h57m46.1463362s) [seg_pos={0 0}, upload=554639]: Segment reached optimal threshold of 13 pieces. Jan 1 00:26:46.275410134 (632h57m46.146493169s) [noise=true , address=77.90.31.19:4841 , piece_id=HZ3R6TCUSLAA5RWH4HQN4QRDVNXG2PRDQLKIQIKDHDEESR6YNQVQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Jan 1 00:26:46.275418400 (632h57m46.146501435s) [noise=true , address=77.90.31.19:4841 , piece_id=HZ3R6TCUSLAA5RWH4HQN4QRDVNXG2PRDQLKIQIKDHDEESR6YNQVQ, seg_pos={0 0}, upload=554639]: Long tail cancellation due to optimal threshold Jan 1 00:26:46.275614693 (632h57m46.146697728s) [noise=true , address=77.90.31.13:4629 , piece_id=IS73FV4O3SAIBO323D7I36BI7C42MNL3ZXTTTFW54YS3Z6HTKO5A, seg_pos={0 0}, upload=554639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:33568->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:33568->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33568->77.90.31.13:4629: use of closed network connection Jan 1 00:26:46.275620565 (632h57m46.1467036s) [noise=true , address=77.90.31.13:4629 , piece_id=IS73FV4O3SAIBO323D7I36BI7C42MNL3ZXTTTFW54YS3Z6HTKO5A, seg_pos={0 0}, upload=554639]: Long tail cancellation due to optimal threshold Jan 1 00:26:46.275644710 (632h57m46.146727745s) [noise=true , address=77.90.31.17:4770 , piece_id=PJ26PIZDCQEXNZGSEGQUQBNLEOCNKARC4UD3DM7WDZOXBLBU47UA, seg_pos={0 0}, upload=554639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 1 00:26:46.275662735 (632h57m46.1467458s) [noise=true , address=77.90.31.17:4770 , piece_id=PJ26PIZDCQEXNZGSEGQUQBNLEOCNKARC4UD3DM7WDZOXBLBU47UA, seg_pos={0 0}, upload=554639]: Long tail cancellation due to optimal threshold Jan 1 00:26:46.275726997 (632h57m46.146810032s) [seg_pos={0 0}, upload=554639]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:26:46.275884958 (632h57m46.146968003s) [upload=554639]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:26:46.275891560 (632h57m46.146974605s) [upload=554639]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:26:46.575173277 (632h57m46.446256312s) [upload=554639]: Done uploading segments... Jan 1 00:26:46.575199937 (632h57m46.446282972s) [upload=554639]: Upload finished. err: <nil> Duration: 6.176109522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X....................X...........X..X.................] duration: 15.225337577s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:27:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 15.225354879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X....................X...........X..X................] duration: 32.231442006s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:27:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 32.231466962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X....X....X......X....X....X.....XXX..X..XX.......XXXX....X] duration: 30.001792132s trace: https://jaeger.dev.storj.io/trace/547ec6510f17298a FAIL (2026-01-01 00:27:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: OKVOAOBKNQKK4EDGZFPOEUZIFQUSMXE75OKRRIXLM3LBUV3KLOZQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: DSUJD6UY2UDUWZOYYT5DG5EYP65NHCYSHINICQEBI45I4XRLKURQ) context deadline exceeded ======LOGS====== Jan 1 00:26:43.639934012 (632h57m43.511017057s) [upload=554639]: Starting upload Jan 1 00:26:43.640013644 (632h57m43.511096689s) [upload=554639]: Done starting upload Jan 1 00:26:43.640065633 (632h57m43.511148668s) [upload=554639]: Uploading segments... Jan 1 00:26:43.640239274 (632h57m43.511322309s) [upload=554639]: Got next segment. Inline: false Jan 1 00:26:43.640275743 (632h57m43.511358778s) [upload=554639]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:26:43.640281293 (632h57m43.511364338s) [upload=554639]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:26:43.887058846 (632h57m43.758141911s) [seg_pos={0 0}, upload=554639]: Begin upload segment... Jan 1 00:26:43.887136624 (632h57m43.758219679s) [seg_pos={0 0}, upload=554639]: Done begin upload segment. Jan 1 00:26:43.887179365 (632h57m43.7582624s) [noise=true , address=145.79.227.52:4445 , piece_id=7NIVFBNHANX2XBLAGD362IZLQODAUXWZ5354YGRDM4QXDAPWGPFQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.887202840 (632h57m43.758285915s) [noise=true , address=145.79.227.15:4876 , piece_id=ZX4FJTXU5MBTN6LDZWP7EZGDPCAQPNWTQGZQ4Q5PKOSRWX42MQ2A, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.888016850 (632h57m43.759099925s) [noise=true , address=77.90.31.17:4770 , piece_id=PJ26PIZDCQEXNZGSEGQUQBNLEOCNKARC4UD3DM7WDZOXBLBU47UA, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.888297224 (632h57m43.759380259s) [noise=true , address=77.90.31.13:4629 , piece_id=IS73FV4O3SAIBO323D7I36BI7C42MNL3ZXTTTFW54YS3Z6HTKO5A, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.888591404 (632h57m43.759674439s) [noise=true , address=145.79.227.11:4732 , piece_id=6X2Z4KXFYZS2JDVEIEYDOTPKDGFUY4UBTHYVZZ6MAYO2UINBEGMQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.888609038 (632h57m43.759692083s) [noise=true , address=45.91.116.10:4081 , piece_id=544HX4WZAGWRHVT5YAM7NOSXSABIR6RTJ35KGDVFLYHLFTGDM4LQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.888838624 (632h57m43.759921659s) [noise=true , address=145.79.227.48:4303 , piece_id=3D2VXRQ4A3WBZSNFROC3A5GFQG53PDZXJZGDPOA7IVE2SYZTKNYA, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889015943 (632h57m43.760098978s) [noise=true , address=45.91.116.5:4300 , piece_id=YPTEYUEENQ4CADNPMAQNQ3VD44MX27VVPJKU7HBLSMQQ2NPDOCTQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889078681 (632h57m43.760161716s) [noise=true , address=45.91.116.12:4228 , piece_id=OKVOAOBKNQKK4EDGZFPOEUZIFQUSMXE75OKRRIXLM3LBUV3KLOZQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889373012 (632h57m43.760456047s) [noise=true , address=145.79.227.14:4839 , piece_id=DMZJWMWXX3T45OBC3XTC5H7VK7Q2YNKLVNCHCPFXFQ5P3W3NAZ4A, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889439960 (632h57m43.760523005s) [noise=true , address=145.79.227.9:4367 , piece_id=MD277W6E5CIWCOARSFHXEF2ADMF6CJUOYFYXGPVJP7F364KWQCFA, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889644059 (632h57m43.760727094s) [noise=true , address=45.91.116.13:4153 , piece_id=DSUJD6UY2UDUWZOYYT5DG5EYP65NHCYSHINICQEBI45I4XRLKURQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.889922669 (632h57m43.761005704s) [noise=true , address=145.79.227.46:4601 , piece_id=Y45NHDBQV7ANRWZGYJX2Q53KOEF3TWHWYFCEOYFSUPKQD662USYQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.890240364 (632h57m43.761323409s) [noise=true , address=77.90.31.19:4841 , piece_id=HZ3R6TCUSLAA5RWH4HQN4QRDVNXG2PRDQLKIQIKDHDEESR6YNQVQ, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.890690932 (632h57m43.761773967s) [noise=true , address=145.79.227.38:4979 , piece_id=BGSLO7XLEC6G3EFOWBZRJW7OJFGLZPHFBGELNBHYMCIM4YIUJP2A, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:43.890998016 (632h57m43.762081051s) [noise=true , address=145.79.227.39:4014 , piece_id=6INN6YZ75DVM3FY5IZV3BHLATS3IRR6RHOLA3G7ZCANKCYZLOXKA, seg_pos={0 0}, upload=554639]: Uploading piece... Jan 1 00:26:45.587750382 (632h57m45.458833417s) [upload=554639]: Next returned nil segment Jan 1 00:26:45.587861183 (632h57m45.458944238s) [upload=554639]: Waiting for error group managing segments... Jan 1 00:26:45.861879976 (632h57m45.732963011s) [noise=true , address=145.79.227.52:4445 , piece_id=7NIVFBNHANX2XBLAGD362IZLQODAUXWZ5354YGRDM4QXDAPWGPFQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.877011187 (632h57m45.748094222s) [noise=true , address=145.79.227.14:4839 , piece_id=DMZJWMWXX3T45OBC3XTC5H7VK7Q2YNKLVNCHCPFXFQ5P3W3NAZ4A, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.877073907 (632h57m45.748156952s) [noise=true , address=145.79.227.48:4303 , piece_id=3D2VXRQ4A3WBZSNFROC3A5GFQG53PDZXJZGDPOA7IVE2SYZTKNYA, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.888489253 (632h57m45.759572288s) [noise=true , address=145.79.227.46:4601 , piece_id=Y45NHDBQV7ANRWZGYJX2Q53KOEF3TWHWYFCEOYFSUPKQD662USYQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.890341650 (632h57m45.761424695s) [noise=true , address=145.79.227.15:4876 , piece_id=ZX4FJTXU5MBTN6LDZWP7EZGDPCAQPNWTQGZQ4Q5PKOSRWX42MQ2A, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.890663152 (632h57m45.761746187s) [noise=true , address=145.79.227.11:4732 , piece_id=6X2Z4KXFYZS2JDVEIEYDOTPKDGFUY4UBTHYVZZ6MAYO2UINBEGMQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.898816437 (632h57m45.769899472s) [noise=true , address=145.79.227.39:4014 , piece_id=6INN6YZ75DVM3FY5IZV3BHLATS3IRR6RHOLA3G7ZCANKCYZLOXKA, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.899081672 (632h57m45.770164707s) [noise=true , address=145.79.227.9:4367 , piece_id=MD277W6E5CIWCOARSFHXEF2ADMF6CJUOYFYXGPVJP7F364KWQCFA, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:45.962848973 (632h57m45.833932008s) [noise=true , address=145.79.227.38:4979 , piece_id=BGSLO7XLEC6G3EFOWBZRJW7OJFGLZPHFBGELNBHYMCIM4YIUJP2A, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:46.190330593 (632h57m46.061413628s) [noise=true , address=45.91.116.12:4228 , piece_id=OKVOAOBKNQKK4EDGZFPOEUZIFQUSMXE75OKRRIXLM3LBUV3KLOZQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:46.239117618 (632h57m46.110200654s) [noise=true , address=45.91.116.13:4153 , piece_id=DSUJD6UY2UDUWZOYYT5DG5EYP65NHCYSHINICQEBI45I4XRLKURQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:46.264420027 (632h57m46.135503062s) [noise=true , address=45.91.116.5:4300 , piece_id=YPTEYUEENQ4CADNPMAQNQ3VD44MX27VVPJKU7HBLSMQQ2NPDOCTQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:46.275234450 (632h57m46.146317485s) [noise=true , address=45.91.116.10:4081 , piece_id=544HX4WZAGWRHVT5YAM7NOSXSABIR6RTJ35KGDVFLYHLFTGDM4LQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: <nil> Jan 1 00:26:46.275253165 (632h57m46.1463362s) [seg_pos={0 0}, upload=554639]: Segment reached optimal threshold of 13 pieces. Jan 1 00:26:46.275410134 (632h57m46.146493169s) [noise=true , address=77.90.31.19:4841 , piece_id=HZ3R6TCUSLAA5RWH4HQN4QRDVNXG2PRDQLKIQIKDHDEESR6YNQVQ, seg_pos={0 0}, upload=554639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Jan 1 00:26:46.275418400 (632h57m46.146501435s) [noise=true , address=77.90.31.19:4841 , piece_id=HZ3R6TCUSLAA5RWH4HQN4QRDVNXG2PRDQLKIQIKDHDEESR6YNQVQ, seg_pos={0 0}, upload=554639]: Long tail cancellation due to optimal threshold Jan 1 00:26:46.275614693 (632h57m46.146697728s) [noise=true , address=77.90.31.13:4629 , piece_id=IS73FV4O3SAIBO323D7I36BI7C42MNL3ZXTTTFW54YS3Z6HTKO5A, seg_pos={0 0}, upload=554639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:33568->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:33568->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33568->77.90.31.13:4629: use of closed network connection Jan 1 00:26:46.275620565 (632h57m46.1467036s) [noise=true , address=77.90.31.13:4629 , piece_id=IS73FV4O3SAIBO323D7I36BI7C42MNL3ZXTTTFW54YS3Z6HTKO5A, seg_pos={0 0}, upload=554639]: Long tail cancellation due to optimal threshold Jan 1 00:26:46.275644710 (632h57m46.146727745s) [noise=true , address=77.90.31.17:4770 , piece_id=PJ26PIZDCQEXNZGSEGQUQBNLEOCNKARC4UD3DM7WDZOXBLBU47UA, seg_pos={0 0}, upload=554639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 1 00:26:46.275662735 (632h57m46.1467458s) [noise=true , address=77.90.31.17:4770 , piece_id=PJ26PIZDCQEXNZGSEGQUQBNLEOCNKARC4UD3DM7WDZOXBLBU47UA, seg_pos={0 0}, upload=554639]: Long tail cancellation due to optimal threshold Jan 1 00:26:46.275726997 (632h57m46.146810032s) [seg_pos={0 0}, upload=554639]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:26:46.275884958 (632h57m46.146968003s) [upload=554639]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:26:46.275891560 (632h57m46.146974605s) [upload=554639]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:26:46.575173277 (632h57m46.446256312s) [upload=554639]: Done uploading segments... Jan 1 00:26:46.575199937 (632h57m46.446282972s) [upload=554639]: Upload finished. err: <nil> Duration: 30.001808292s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X....................X...........X..X...............] duration: 17.090142063s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:26:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 17.090258204s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X....X....X......X....X....X.....XXX..X..XX.......XXXX....] duration: 5.866480777s trace: https://jaeger.dev.storj.io/trace/40198cd83b11a0ac FAIL (2026-01-01 00:26:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 1 00:22:12.667100044 (632h53m12.538183089s) [upload=554577]: Starting upload Jan 1 00:22:12.667205795 (632h53m12.53828884s) [upload=554577]: Done starting upload Jan 1 00:22:12.667346473 (632h53m12.538429518s) [upload=554577]: Uploading segments... Jan 1 00:22:12.668198986 (632h53m12.539282031s) [upload=554577]: Got next segment. Inline: false Jan 1 00:22:12.668249411 (632h53m12.539332446s) [upload=554577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:22:12.668254341 (632h53m12.539337376s) [upload=554577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:22:12.914286884 (632h53m12.785369919s) [seg_pos={0 0}, upload=554577]: Begin upload segment... Jan 1 00:22:12.914345125 (632h53m12.78542816s) [seg_pos={0 0}, upload=554577]: Done begin upload segment. Jan 1 00:22:12.914399499 (632h53m12.785482534s) [noise=true , address=45.91.116.7:4010 , piece_id=MHPDUBDLGIV5GRETMRGFDKD52KWSRD4AHMQCIJQ44GMAIEGOYCKA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.915934121 (632h53m12.787017156s) [noise=true , address=145.79.227.2:4113 , piece_id=YOODB72PJN6HKHIZK5SEVLSPRLKOWQT2R36MBUE3IDTOAKLZRNOA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.916271343 (632h53m12.787354378s) [noise=true , address=145.79.227.52:4445 , piece_id=TYB3H7ZEA5WVDT7F2REXCGEWYBF5TKL7PP5RQLWBHIDQWTFWOBVQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.916693157 (632h53m12.787776202s) [noise=true , address=77.90.31.10:4520 , piece_id=WJUSK2HNXXWXO2ZOFEP3J5AOOXFLKOL6IVZQFHWTRKT6A6GL5ZNA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.917080064 (632h53m12.788163099s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQA7G3YWAIKSYCDZKEO2U2ZHMDDUP4HODXJ3ZNIJHHF3ZHMJ7AQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.917550629 (632h53m12.788633674s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.918792294 (632h53m12.789875339s) [noise=true , address=77.90.31.17:4773 , piece_id=JYUVRRIMAU2OSSH5LCRRMUHHUJAFXZE3VYK5QUO4YBUXKV46ODQQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.918905790 (632h53m12.789988835s) [noise=true , address=145.79.227.26:4655 , piece_id=GMUHN5KKFONXNLMOAI5BCQJ36MR6RMSHIGVDPW4RUVZICX6T32LQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920095385 (632h53m12.79117842s) [noise=true , address=145.79.227.27:4690 , piece_id=FB64RYKQ6CXCF2L6A55XIRUBZJRQV3VQZOSK4WAWN77CKFVFLJ3Q, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920153916 (632h53m12.791236951s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920423229 (632h53m12.791506264s) [noise=true , address=145.79.227.50:4375 , piece_id=DQLX7VHWDBOFHXAJSIO5YDVUMWA6CGMI7ZRVV3AD3UTJWROH4FOQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.921453851 (632h53m12.792536896s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.921737862 (632h53m12.792820897s) [noise=true , address=145.79.227.16:4912 , piece_id=XX7ZEGC4ZZGKD3YVQYC2BSKEYJKFGGNZIF63SL4ARF3XTCUJPWMQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.923167494 (632h53m12.794250529s) [noise=true , address=145.79.227.4:4186 , piece_id=QCMIEG2NDHYFCXIR3CASGUEJEXKRDDQPU6X46E3YDOSNEWGWP7LQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.923410737 (632h53m12.794493782s) [noise=true , address=145.79.227.5:4221 , piece_id=SWHMPK7JYIVDPJJYXIM5YS5YGD3QNOF5GO57NK3TA226YPUWBEWA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.924706354 (632h53m12.795789389s) [noise=true , address=145.79.227.18:4007 , piece_id=TPBTA7BC332TCQVFWGEHW5IIUPX7VWVUIXLNEU6Y2GYR6UUV27TQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:14.630887525 (632h53m14.50197056s) [upload=554577]: Next returned nil segment Jan 1 00:22:14.630916360 (632h53m14.501999395s) [upload=554577]: Waiting for error group managing segments... Jan 1 00:22:14.909834801 (632h53m14.780917847s) [noise=true , address=145.79.227.2:4113 , piece_id=YOODB72PJN6HKHIZK5SEVLSPRLKOWQT2R36MBUE3IDTOAKLZRNOA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.921054485 (632h53m14.79213752s) [noise=true , address=145.79.227.5:4221 , piece_id=SWHMPK7JYIVDPJJYXIM5YS5YGD3QNOF5GO57NK3TA226YPUWBEWA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.921424640 (632h53m14.792507686s) [noise=true , address=145.79.227.52:4445 , piece_id=TYB3H7ZEA5WVDT7F2REXCGEWYBF5TKL7PP5RQLWBHIDQWTFWOBVQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.923275004 (632h53m14.794358039s) [noise=true , address=145.79.227.4:4186 , piece_id=QCMIEG2NDHYFCXIR3CASGUEJEXKRDDQPU6X46E3YDOSNEWGWP7LQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.934454711 (632h53m14.805537746s) [noise=true , address=145.79.227.18:4007 , piece_id=TPBTA7BC332TCQVFWGEHW5IIUPX7VWVUIXLNEU6Y2GYR6UUV27TQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.946354569 (632h53m14.817437604s) [noise=true , address=145.79.227.27:4690 , piece_id=FB64RYKQ6CXCF2L6A55XIRUBZJRQV3VQZOSK4WAWN77CKFVFLJ3Q, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.947893469 (632h53m14.818976504s) [noise=true , address=145.79.227.26:4655 , piece_id=GMUHN5KKFONXNLMOAI5BCQJ36MR6RMSHIGVDPW4RUVZICX6T32LQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.951993707 (632h53m14.823076752s) [noise=true , address=145.79.227.50:4375 , piece_id=DQLX7VHWDBOFHXAJSIO5YDVUMWA6CGMI7ZRVV3AD3UTJWROH4FOQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.983034743 (632h53m14.854117788s) [noise=true , address=145.79.227.16:4912 , piece_id=XX7ZEGC4ZZGKD3YVQYC2BSKEYJKFGGNZIF63SL4ARF3XTCUJPWMQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.203948107 (632h53m15.075031152s) [noise=true , address=45.91.116.7:4010 , piece_id=MHPDUBDLGIV5GRETMRGFDKD52KWSRD4AHMQCIJQ44GMAIEGOYCKA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.216064908 (632h53m15.087147944s) [noise=true , address=77.90.31.17:4773 , piece_id=JYUVRRIMAU2OSSH5LCRRMUHHUJAFXZE3VYK5QUO4YBUXKV46ODQQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.216273646 (632h53m15.087356691s) [noise=true , address=77.90.31.10:4520 , piece_id=WJUSK2HNXXWXO2ZOFEP3J5AOOXFLKOL6IVZQFHWTRKT6A6GL5ZNA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.342047284 (632h53m15.213130319s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQA7G3YWAIKSYCDZKEO2U2ZHMDDUP4HODXJ3ZNIJHHF3ZHMJ7AQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.342084845 (632h53m15.21316789s) [seg_pos={0 0}, upload=554577]: Segment reached optimal threshold of 13 pieces. Jan 1 00:22:15.342300776 (632h53m15.213383821s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Jan 1 00:22:15.342308010 (632h53m15.213391045s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342356141 (632h53m15.213439186s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 1 00:22:15.342372963 (632h53m15.213456008s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342726086 (632h53m15.213809121s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection Jan 1 00:22:15.342735083 (632h53m15.213818128s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342757906 (632h53m15.213840941s) [seg_pos={0 0}, upload=554577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:22:15.342836896 (632h53m15.213919932s) [upload=554577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:22:15.342853028 (632h53m15.213936063s) [upload=554577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:22:15.639323159 (632h53m15.510406195s) [upload=554577]: Done uploading segments... Jan 1 00:22:15.639343898 (632h53m15.510426933s) [upload=554577]: Upload finished. err: <nil> Duration: 5.866512837s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X....................X...........X..X..............] duration: 22.540819693s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:25:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 22.540880479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X....X....X......X....X....X.....XXX..X..XX.......XXXX...] duration: 6.188906774s trace: https://jaeger.dev.storj.io/trace/40198cd83b11a0ac FAIL (2026-01-01 00:25:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 1 00:22:12.667100044 (632h53m12.538183089s) [upload=554577]: Starting upload Jan 1 00:22:12.667205795 (632h53m12.53828884s) [upload=554577]: Done starting upload Jan 1 00:22:12.667346473 (632h53m12.538429518s) [upload=554577]: Uploading segments... Jan 1 00:22:12.668198986 (632h53m12.539282031s) [upload=554577]: Got next segment. Inline: false Jan 1 00:22:12.668249411 (632h53m12.539332446s) [upload=554577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:22:12.668254341 (632h53m12.539337376s) [upload=554577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:22:12.914286884 (632h53m12.785369919s) [seg_pos={0 0}, upload=554577]: Begin upload segment... Jan 1 00:22:12.914345125 (632h53m12.78542816s) [seg_pos={0 0}, upload=554577]: Done begin upload segment. Jan 1 00:22:12.914399499 (632h53m12.785482534s) [noise=true , address=45.91.116.7:4010 , piece_id=MHPDUBDLGIV5GRETMRGFDKD52KWSRD4AHMQCIJQ44GMAIEGOYCKA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.915934121 (632h53m12.787017156s) [noise=true , address=145.79.227.2:4113 , piece_id=YOODB72PJN6HKHIZK5SEVLSPRLKOWQT2R36MBUE3IDTOAKLZRNOA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.916271343 (632h53m12.787354378s) [noise=true , address=145.79.227.52:4445 , piece_id=TYB3H7ZEA5WVDT7F2REXCGEWYBF5TKL7PP5RQLWBHIDQWTFWOBVQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.916693157 (632h53m12.787776202s) [noise=true , address=77.90.31.10:4520 , piece_id=WJUSK2HNXXWXO2ZOFEP3J5AOOXFLKOL6IVZQFHWTRKT6A6GL5ZNA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.917080064 (632h53m12.788163099s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQA7G3YWAIKSYCDZKEO2U2ZHMDDUP4HODXJ3ZNIJHHF3ZHMJ7AQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.917550629 (632h53m12.788633674s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.918792294 (632h53m12.789875339s) [noise=true , address=77.90.31.17:4773 , piece_id=JYUVRRIMAU2OSSH5LCRRMUHHUJAFXZE3VYK5QUO4YBUXKV46ODQQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.918905790 (632h53m12.789988835s) [noise=true , address=145.79.227.26:4655 , piece_id=GMUHN5KKFONXNLMOAI5BCQJ36MR6RMSHIGVDPW4RUVZICX6T32LQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920095385 (632h53m12.79117842s) [noise=true , address=145.79.227.27:4690 , piece_id=FB64RYKQ6CXCF2L6A55XIRUBZJRQV3VQZOSK4WAWN77CKFVFLJ3Q, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920153916 (632h53m12.791236951s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920423229 (632h53m12.791506264s) [noise=true , address=145.79.227.50:4375 , piece_id=DQLX7VHWDBOFHXAJSIO5YDVUMWA6CGMI7ZRVV3AD3UTJWROH4FOQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.921453851 (632h53m12.792536896s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.921737862 (632h53m12.792820897s) [noise=true , address=145.79.227.16:4912 , piece_id=XX7ZEGC4ZZGKD3YVQYC2BSKEYJKFGGNZIF63SL4ARF3XTCUJPWMQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.923167494 (632h53m12.794250529s) [noise=true , address=145.79.227.4:4186 , piece_id=QCMIEG2NDHYFCXIR3CASGUEJEXKRDDQPU6X46E3YDOSNEWGWP7LQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.923410737 (632h53m12.794493782s) [noise=true , address=145.79.227.5:4221 , piece_id=SWHMPK7JYIVDPJJYXIM5YS5YGD3QNOF5GO57NK3TA226YPUWBEWA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.924706354 (632h53m12.795789389s) [noise=true , address=145.79.227.18:4007 , piece_id=TPBTA7BC332TCQVFWGEHW5IIUPX7VWVUIXLNEU6Y2GYR6UUV27TQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:14.630887525 (632h53m14.50197056s) [upload=554577]: Next returned nil segment Jan 1 00:22:14.630916360 (632h53m14.501999395s) [upload=554577]: Waiting for error group managing segments... Jan 1 00:22:14.909834801 (632h53m14.780917847s) [noise=true , address=145.79.227.2:4113 , piece_id=YOODB72PJN6HKHIZK5SEVLSPRLKOWQT2R36MBUE3IDTOAKLZRNOA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.921054485 (632h53m14.79213752s) [noise=true , address=145.79.227.5:4221 , piece_id=SWHMPK7JYIVDPJJYXIM5YS5YGD3QNOF5GO57NK3TA226YPUWBEWA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.921424640 (632h53m14.792507686s) [noise=true , address=145.79.227.52:4445 , piece_id=TYB3H7ZEA5WVDT7F2REXCGEWYBF5TKL7PP5RQLWBHIDQWTFWOBVQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.923275004 (632h53m14.794358039s) [noise=true , address=145.79.227.4:4186 , piece_id=QCMIEG2NDHYFCXIR3CASGUEJEXKRDDQPU6X46E3YDOSNEWGWP7LQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.934454711 (632h53m14.805537746s) [noise=true , address=145.79.227.18:4007 , piece_id=TPBTA7BC332TCQVFWGEHW5IIUPX7VWVUIXLNEU6Y2GYR6UUV27TQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.946354569 (632h53m14.817437604s) [noise=true , address=145.79.227.27:4690 , piece_id=FB64RYKQ6CXCF2L6A55XIRUBZJRQV3VQZOSK4WAWN77CKFVFLJ3Q, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.947893469 (632h53m14.818976504s) [noise=true , address=145.79.227.26:4655 , piece_id=GMUHN5KKFONXNLMOAI5BCQJ36MR6RMSHIGVDPW4RUVZICX6T32LQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.951993707 (632h53m14.823076752s) [noise=true , address=145.79.227.50:4375 , piece_id=DQLX7VHWDBOFHXAJSIO5YDVUMWA6CGMI7ZRVV3AD3UTJWROH4FOQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.983034743 (632h53m14.854117788s) [noise=true , address=145.79.227.16:4912 , piece_id=XX7ZEGC4ZZGKD3YVQYC2BSKEYJKFGGNZIF63SL4ARF3XTCUJPWMQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.203948107 (632h53m15.075031152s) [noise=true , address=45.91.116.7:4010 , piece_id=MHPDUBDLGIV5GRETMRGFDKD52KWSRD4AHMQCIJQ44GMAIEGOYCKA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.216064908 (632h53m15.087147944s) [noise=true , address=77.90.31.17:4773 , piece_id=JYUVRRIMAU2OSSH5LCRRMUHHUJAFXZE3VYK5QUO4YBUXKV46ODQQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.216273646 (632h53m15.087356691s) [noise=true , address=77.90.31.10:4520 , piece_id=WJUSK2HNXXWXO2ZOFEP3J5AOOXFLKOL6IVZQFHWTRKT6A6GL5ZNA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.342047284 (632h53m15.213130319s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQA7G3YWAIKSYCDZKEO2U2ZHMDDUP4HODXJ3ZNIJHHF3ZHMJ7AQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.342084845 (632h53m15.21316789s) [seg_pos={0 0}, upload=554577]: Segment reached optimal threshold of 13 pieces. Jan 1 00:22:15.342300776 (632h53m15.213383821s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Jan 1 00:22:15.342308010 (632h53m15.213391045s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342356141 (632h53m15.213439186s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 1 00:22:15.342372963 (632h53m15.213456008s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342726086 (632h53m15.213809121s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection Jan 1 00:22:15.342735083 (632h53m15.213818128s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342757906 (632h53m15.213840941s) [seg_pos={0 0}, upload=554577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:22:15.342836896 (632h53m15.213919932s) [upload=554577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:22:15.342853028 (632h53m15.213936063s) [upload=554577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:22:15.639323159 (632h53m15.510406195s) [upload=554577]: Done uploading segments... Jan 1 00:22:15.639343898 (632h53m15.510426933s) [upload=554577]: Upload finished. err: <nil> Duration: 6.188934777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X....................X...........X..X.............] duration: 19.391342443s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:24:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 19.391366309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X....X....X......X....X....X.....XXX..X..XX.......XXXX..] duration: 6.234166737s trace: https://jaeger.dev.storj.io/trace/40198cd83b11a0ac FAIL (2026-01-01 00:24:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 1 00:22:12.667100044 (632h53m12.538183089s) [upload=554577]: Starting upload Jan 1 00:22:12.667205795 (632h53m12.53828884s) [upload=554577]: Done starting upload Jan 1 00:22:12.667346473 (632h53m12.538429518s) [upload=554577]: Uploading segments... Jan 1 00:22:12.668198986 (632h53m12.539282031s) [upload=554577]: Got next segment. Inline: false Jan 1 00:22:12.668249411 (632h53m12.539332446s) [upload=554577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:22:12.668254341 (632h53m12.539337376s) [upload=554577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:22:12.914286884 (632h53m12.785369919s) [seg_pos={0 0}, upload=554577]: Begin upload segment... Jan 1 00:22:12.914345125 (632h53m12.78542816s) [seg_pos={0 0}, upload=554577]: Done begin upload segment. Jan 1 00:22:12.914399499 (632h53m12.785482534s) [noise=true , address=45.91.116.7:4010 , piece_id=MHPDUBDLGIV5GRETMRGFDKD52KWSRD4AHMQCIJQ44GMAIEGOYCKA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.915934121 (632h53m12.787017156s) [noise=true , address=145.79.227.2:4113 , piece_id=YOODB72PJN6HKHIZK5SEVLSPRLKOWQT2R36MBUE3IDTOAKLZRNOA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.916271343 (632h53m12.787354378s) [noise=true , address=145.79.227.52:4445 , piece_id=TYB3H7ZEA5WVDT7F2REXCGEWYBF5TKL7PP5RQLWBHIDQWTFWOBVQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.916693157 (632h53m12.787776202s) [noise=true , address=77.90.31.10:4520 , piece_id=WJUSK2HNXXWXO2ZOFEP3J5AOOXFLKOL6IVZQFHWTRKT6A6GL5ZNA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.917080064 (632h53m12.788163099s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQA7G3YWAIKSYCDZKEO2U2ZHMDDUP4HODXJ3ZNIJHHF3ZHMJ7AQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.917550629 (632h53m12.788633674s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.918792294 (632h53m12.789875339s) [noise=true , address=77.90.31.17:4773 , piece_id=JYUVRRIMAU2OSSH5LCRRMUHHUJAFXZE3VYK5QUO4YBUXKV46ODQQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.918905790 (632h53m12.789988835s) [noise=true , address=145.79.227.26:4655 , piece_id=GMUHN5KKFONXNLMOAI5BCQJ36MR6RMSHIGVDPW4RUVZICX6T32LQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920095385 (632h53m12.79117842s) [noise=true , address=145.79.227.27:4690 , piece_id=FB64RYKQ6CXCF2L6A55XIRUBZJRQV3VQZOSK4WAWN77CKFVFLJ3Q, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920153916 (632h53m12.791236951s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920423229 (632h53m12.791506264s) [noise=true , address=145.79.227.50:4375 , piece_id=DQLX7VHWDBOFHXAJSIO5YDVUMWA6CGMI7ZRVV3AD3UTJWROH4FOQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.921453851 (632h53m12.792536896s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.921737862 (632h53m12.792820897s) [noise=true , address=145.79.227.16:4912 , piece_id=XX7ZEGC4ZZGKD3YVQYC2BSKEYJKFGGNZIF63SL4ARF3XTCUJPWMQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.923167494 (632h53m12.794250529s) [noise=true , address=145.79.227.4:4186 , piece_id=QCMIEG2NDHYFCXIR3CASGUEJEXKRDDQPU6X46E3YDOSNEWGWP7LQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.923410737 (632h53m12.794493782s) [noise=true , address=145.79.227.5:4221 , piece_id=SWHMPK7JYIVDPJJYXIM5YS5YGD3QNOF5GO57NK3TA226YPUWBEWA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.924706354 (632h53m12.795789389s) [noise=true , address=145.79.227.18:4007 , piece_id=TPBTA7BC332TCQVFWGEHW5IIUPX7VWVUIXLNEU6Y2GYR6UUV27TQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:14.630887525 (632h53m14.50197056s) [upload=554577]: Next returned nil segment Jan 1 00:22:14.630916360 (632h53m14.501999395s) [upload=554577]: Waiting for error group managing segments... Jan 1 00:22:14.909834801 (632h53m14.780917847s) [noise=true , address=145.79.227.2:4113 , piece_id=YOODB72PJN6HKHIZK5SEVLSPRLKOWQT2R36MBUE3IDTOAKLZRNOA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.921054485 (632h53m14.79213752s) [noise=true , address=145.79.227.5:4221 , piece_id=SWHMPK7JYIVDPJJYXIM5YS5YGD3QNOF5GO57NK3TA226YPUWBEWA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.921424640 (632h53m14.792507686s) [noise=true , address=145.79.227.52:4445 , piece_id=TYB3H7ZEA5WVDT7F2REXCGEWYBF5TKL7PP5RQLWBHIDQWTFWOBVQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.923275004 (632h53m14.794358039s) [noise=true , address=145.79.227.4:4186 , piece_id=QCMIEG2NDHYFCXIR3CASGUEJEXKRDDQPU6X46E3YDOSNEWGWP7LQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.934454711 (632h53m14.805537746s) [noise=true , address=145.79.227.18:4007 , piece_id=TPBTA7BC332TCQVFWGEHW5IIUPX7VWVUIXLNEU6Y2GYR6UUV27TQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.946354569 (632h53m14.817437604s) [noise=true , address=145.79.227.27:4690 , piece_id=FB64RYKQ6CXCF2L6A55XIRUBZJRQV3VQZOSK4WAWN77CKFVFLJ3Q, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.947893469 (632h53m14.818976504s) [noise=true , address=145.79.227.26:4655 , piece_id=GMUHN5KKFONXNLMOAI5BCQJ36MR6RMSHIGVDPW4RUVZICX6T32LQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.951993707 (632h53m14.823076752s) [noise=true , address=145.79.227.50:4375 , piece_id=DQLX7VHWDBOFHXAJSIO5YDVUMWA6CGMI7ZRVV3AD3UTJWROH4FOQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.983034743 (632h53m14.854117788s) [noise=true , address=145.79.227.16:4912 , piece_id=XX7ZEGC4ZZGKD3YVQYC2BSKEYJKFGGNZIF63SL4ARF3XTCUJPWMQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.203948107 (632h53m15.075031152s) [noise=true , address=45.91.116.7:4010 , piece_id=MHPDUBDLGIV5GRETMRGFDKD52KWSRD4AHMQCIJQ44GMAIEGOYCKA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.216064908 (632h53m15.087147944s) [noise=true , address=77.90.31.17:4773 , piece_id=JYUVRRIMAU2OSSH5LCRRMUHHUJAFXZE3VYK5QUO4YBUXKV46ODQQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.216273646 (632h53m15.087356691s) [noise=true , address=77.90.31.10:4520 , piece_id=WJUSK2HNXXWXO2ZOFEP3J5AOOXFLKOL6IVZQFHWTRKT6A6GL5ZNA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.342047284 (632h53m15.213130319s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQA7G3YWAIKSYCDZKEO2U2ZHMDDUP4HODXJ3ZNIJHHF3ZHMJ7AQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.342084845 (632h53m15.21316789s) [seg_pos={0 0}, upload=554577]: Segment reached optimal threshold of 13 pieces. Jan 1 00:22:15.342300776 (632h53m15.213383821s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Jan 1 00:22:15.342308010 (632h53m15.213391045s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342356141 (632h53m15.213439186s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 1 00:22:15.342372963 (632h53m15.213456008s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342726086 (632h53m15.213809121s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection Jan 1 00:22:15.342735083 (632h53m15.213818128s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342757906 (632h53m15.213840941s) [seg_pos={0 0}, upload=554577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:22:15.342836896 (632h53m15.213919932s) [upload=554577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:22:15.342853028 (632h53m15.213936063s) [upload=554577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:22:15.639323159 (632h53m15.510406195s) [upload=554577]: Done uploading segments... Jan 1 00:22:15.639343898 (632h53m15.510426933s) [upload=554577]: Upload finished. err: <nil> Duration: 6.234195091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X....X....X......X....X....X.....XXX..X..XX.......XXXX.] duration: 5.995504459s trace: https://jaeger.dev.storj.io/trace/40198cd83b11a0ac FAIL (2026-01-01 00:23:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 1 00:22:12.667100044 (632h53m12.538183089s) [upload=554577]: Starting upload Jan 1 00:22:12.667205795 (632h53m12.53828884s) [upload=554577]: Done starting upload Jan 1 00:22:12.667346473 (632h53m12.538429518s) [upload=554577]: Uploading segments... Jan 1 00:22:12.668198986 (632h53m12.539282031s) [upload=554577]: Got next segment. Inline: false Jan 1 00:22:12.668249411 (632h53m12.539332446s) [upload=554577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:22:12.668254341 (632h53m12.539337376s) [upload=554577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:22:12.914286884 (632h53m12.785369919s) [seg_pos={0 0}, upload=554577]: Begin upload segment... Jan 1 00:22:12.914345125 (632h53m12.78542816s) [seg_pos={0 0}, upload=554577]: Done begin upload segment. Jan 1 00:22:12.914399499 (632h53m12.785482534s) [noise=true , address=45.91.116.7:4010 , piece_id=MHPDUBDLGIV5GRETMRGFDKD52KWSRD4AHMQCIJQ44GMAIEGOYCKA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.915934121 (632h53m12.787017156s) [noise=true , address=145.79.227.2:4113 , piece_id=YOODB72PJN6HKHIZK5SEVLSPRLKOWQT2R36MBUE3IDTOAKLZRNOA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.916271343 (632h53m12.787354378s) [noise=true , address=145.79.227.52:4445 , piece_id=TYB3H7ZEA5WVDT7F2REXCGEWYBF5TKL7PP5RQLWBHIDQWTFWOBVQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.916693157 (632h53m12.787776202s) [noise=true , address=77.90.31.10:4520 , piece_id=WJUSK2HNXXWXO2ZOFEP3J5AOOXFLKOL6IVZQFHWTRKT6A6GL5ZNA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.917080064 (632h53m12.788163099s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQA7G3YWAIKSYCDZKEO2U2ZHMDDUP4HODXJ3ZNIJHHF3ZHMJ7AQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.917550629 (632h53m12.788633674s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.918792294 (632h53m12.789875339s) [noise=true , address=77.90.31.17:4773 , piece_id=JYUVRRIMAU2OSSH5LCRRMUHHUJAFXZE3VYK5QUO4YBUXKV46ODQQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.918905790 (632h53m12.789988835s) [noise=true , address=145.79.227.26:4655 , piece_id=GMUHN5KKFONXNLMOAI5BCQJ36MR6RMSHIGVDPW4RUVZICX6T32LQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920095385 (632h53m12.79117842s) [noise=true , address=145.79.227.27:4690 , piece_id=FB64RYKQ6CXCF2L6A55XIRUBZJRQV3VQZOSK4WAWN77CKFVFLJ3Q, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920153916 (632h53m12.791236951s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920423229 (632h53m12.791506264s) [noise=true , address=145.79.227.50:4375 , piece_id=DQLX7VHWDBOFHXAJSIO5YDVUMWA6CGMI7ZRVV3AD3UTJWROH4FOQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.921453851 (632h53m12.792536896s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.921737862 (632h53m12.792820897s) [noise=true , address=145.79.227.16:4912 , piece_id=XX7ZEGC4ZZGKD3YVQYC2BSKEYJKFGGNZIF63SL4ARF3XTCUJPWMQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.923167494 (632h53m12.794250529s) [noise=true , address=145.79.227.4:4186 , piece_id=QCMIEG2NDHYFCXIR3CASGUEJEXKRDDQPU6X46E3YDOSNEWGWP7LQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.923410737 (632h53m12.794493782s) [noise=true , address=145.79.227.5:4221 , piece_id=SWHMPK7JYIVDPJJYXIM5YS5YGD3QNOF5GO57NK3TA226YPUWBEWA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.924706354 (632h53m12.795789389s) [noise=true , address=145.79.227.18:4007 , piece_id=TPBTA7BC332TCQVFWGEHW5IIUPX7VWVUIXLNEU6Y2GYR6UUV27TQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:14.630887525 (632h53m14.50197056s) [upload=554577]: Next returned nil segment Jan 1 00:22:14.630916360 (632h53m14.501999395s) [upload=554577]: Waiting for error group managing segments... Jan 1 00:22:14.909834801 (632h53m14.780917847s) [noise=true , address=145.79.227.2:4113 , piece_id=YOODB72PJN6HKHIZK5SEVLSPRLKOWQT2R36MBUE3IDTOAKLZRNOA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.921054485 (632h53m14.79213752s) [noise=true , address=145.79.227.5:4221 , piece_id=SWHMPK7JYIVDPJJYXIM5YS5YGD3QNOF5GO57NK3TA226YPUWBEWA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.921424640 (632h53m14.792507686s) [noise=true , address=145.79.227.52:4445 , piece_id=TYB3H7ZEA5WVDT7F2REXCGEWYBF5TKL7PP5RQLWBHIDQWTFWOBVQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.923275004 (632h53m14.794358039s) [noise=true , address=145.79.227.4:4186 , piece_id=QCMIEG2NDHYFCXIR3CASGUEJEXKRDDQPU6X46E3YDOSNEWGWP7LQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.934454711 (632h53m14.805537746s) [noise=true , address=145.79.227.18:4007 , piece_id=TPBTA7BC332TCQVFWGEHW5IIUPX7VWVUIXLNEU6Y2GYR6UUV27TQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.946354569 (632h53m14.817437604s) [noise=true , address=145.79.227.27:4690 , piece_id=FB64RYKQ6CXCF2L6A55XIRUBZJRQV3VQZOSK4WAWN77CKFVFLJ3Q, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.947893469 (632h53m14.818976504s) [noise=true , address=145.79.227.26:4655 , piece_id=GMUHN5KKFONXNLMOAI5BCQJ36MR6RMSHIGVDPW4RUVZICX6T32LQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.951993707 (632h53m14.823076752s) [noise=true , address=145.79.227.50:4375 , piece_id=DQLX7VHWDBOFHXAJSIO5YDVUMWA6CGMI7ZRVV3AD3UTJWROH4FOQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.983034743 (632h53m14.854117788s) [noise=true , address=145.79.227.16:4912 , piece_id=XX7ZEGC4ZZGKD3YVQYC2BSKEYJKFGGNZIF63SL4ARF3XTCUJPWMQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.203948107 (632h53m15.075031152s) [noise=true , address=45.91.116.7:4010 , piece_id=MHPDUBDLGIV5GRETMRGFDKD52KWSRD4AHMQCIJQ44GMAIEGOYCKA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.216064908 (632h53m15.087147944s) [noise=true , address=77.90.31.17:4773 , piece_id=JYUVRRIMAU2OSSH5LCRRMUHHUJAFXZE3VYK5QUO4YBUXKV46ODQQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.216273646 (632h53m15.087356691s) [noise=true , address=77.90.31.10:4520 , piece_id=WJUSK2HNXXWXO2ZOFEP3J5AOOXFLKOL6IVZQFHWTRKT6A6GL5ZNA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.342047284 (632h53m15.213130319s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQA7G3YWAIKSYCDZKEO2U2ZHMDDUP4HODXJ3ZNIJHHF3ZHMJ7AQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.342084845 (632h53m15.21316789s) [seg_pos={0 0}, upload=554577]: Segment reached optimal threshold of 13 pieces. Jan 1 00:22:15.342300776 (632h53m15.213383821s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Jan 1 00:22:15.342308010 (632h53m15.213391045s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342356141 (632h53m15.213439186s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 1 00:22:15.342372963 (632h53m15.213456008s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342726086 (632h53m15.213809121s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection Jan 1 00:22:15.342735083 (632h53m15.213818128s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342757906 (632h53m15.213840941s) [seg_pos={0 0}, upload=554577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:22:15.342836896 (632h53m15.213919932s) [upload=554577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:22:15.342853028 (632h53m15.213936063s) [upload=554577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:22:15.639323159 (632h53m15.510406195s) [upload=554577]: Done uploading segments... Jan 1 00:22:15.639343898 (632h53m15.510426933s) [upload=554577]: Upload finished. err: <nil> Duration: 5.995529737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X....................X...........X..X............] duration: 18.638666848s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:23:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 18.638698117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X....X....X......X....X....X.....XXX..X..XX.......XXXX] duration: 30.003227614s trace: https://jaeger.dev.storj.io/trace/40198cd83b11a0ac FAIL (2026-01-01 00:22:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 1 00:22:12.667100044 (632h53m12.538183089s) [upload=554577]: Starting upload Jan 1 00:22:12.667205795 (632h53m12.53828884s) [upload=554577]: Done starting upload Jan 1 00:22:12.667346473 (632h53m12.538429518s) [upload=554577]: Uploading segments... Jan 1 00:22:12.668198986 (632h53m12.539282031s) [upload=554577]: Got next segment. Inline: false Jan 1 00:22:12.668249411 (632h53m12.539332446s) [upload=554577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:22:12.668254341 (632h53m12.539337376s) [upload=554577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:22:12.914286884 (632h53m12.785369919s) [seg_pos={0 0}, upload=554577]: Begin upload segment... Jan 1 00:22:12.914345125 (632h53m12.78542816s) [seg_pos={0 0}, upload=554577]: Done begin upload segment. Jan 1 00:22:12.914399499 (632h53m12.785482534s) [noise=true , address=45.91.116.7:4010 , piece_id=MHPDUBDLGIV5GRETMRGFDKD52KWSRD4AHMQCIJQ44GMAIEGOYCKA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.915934121 (632h53m12.787017156s) [noise=true , address=145.79.227.2:4113 , piece_id=YOODB72PJN6HKHIZK5SEVLSPRLKOWQT2R36MBUE3IDTOAKLZRNOA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.916271343 (632h53m12.787354378s) [noise=true , address=145.79.227.52:4445 , piece_id=TYB3H7ZEA5WVDT7F2REXCGEWYBF5TKL7PP5RQLWBHIDQWTFWOBVQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.916693157 (632h53m12.787776202s) [noise=true , address=77.90.31.10:4520 , piece_id=WJUSK2HNXXWXO2ZOFEP3J5AOOXFLKOL6IVZQFHWTRKT6A6GL5ZNA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.917080064 (632h53m12.788163099s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQA7G3YWAIKSYCDZKEO2U2ZHMDDUP4HODXJ3ZNIJHHF3ZHMJ7AQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.917550629 (632h53m12.788633674s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.918792294 (632h53m12.789875339s) [noise=true , address=77.90.31.17:4773 , piece_id=JYUVRRIMAU2OSSH5LCRRMUHHUJAFXZE3VYK5QUO4YBUXKV46ODQQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.918905790 (632h53m12.789988835s) [noise=true , address=145.79.227.26:4655 , piece_id=GMUHN5KKFONXNLMOAI5BCQJ36MR6RMSHIGVDPW4RUVZICX6T32LQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920095385 (632h53m12.79117842s) [noise=true , address=145.79.227.27:4690 , piece_id=FB64RYKQ6CXCF2L6A55XIRUBZJRQV3VQZOSK4WAWN77CKFVFLJ3Q, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920153916 (632h53m12.791236951s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.920423229 (632h53m12.791506264s) [noise=true , address=145.79.227.50:4375 , piece_id=DQLX7VHWDBOFHXAJSIO5YDVUMWA6CGMI7ZRVV3AD3UTJWROH4FOQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.921453851 (632h53m12.792536896s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.921737862 (632h53m12.792820897s) [noise=true , address=145.79.227.16:4912 , piece_id=XX7ZEGC4ZZGKD3YVQYC2BSKEYJKFGGNZIF63SL4ARF3XTCUJPWMQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.923167494 (632h53m12.794250529s) [noise=true , address=145.79.227.4:4186 , piece_id=QCMIEG2NDHYFCXIR3CASGUEJEXKRDDQPU6X46E3YDOSNEWGWP7LQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.923410737 (632h53m12.794493782s) [noise=true , address=145.79.227.5:4221 , piece_id=SWHMPK7JYIVDPJJYXIM5YS5YGD3QNOF5GO57NK3TA226YPUWBEWA, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:12.924706354 (632h53m12.795789389s) [noise=true , address=145.79.227.18:4007 , piece_id=TPBTA7BC332TCQVFWGEHW5IIUPX7VWVUIXLNEU6Y2GYR6UUV27TQ, seg_pos={0 0}, upload=554577]: Uploading piece... Jan 1 00:22:14.630887525 (632h53m14.50197056s) [upload=554577]: Next returned nil segment Jan 1 00:22:14.630916360 (632h53m14.501999395s) [upload=554577]: Waiting for error group managing segments... Jan 1 00:22:14.909834801 (632h53m14.780917847s) [noise=true , address=145.79.227.2:4113 , piece_id=YOODB72PJN6HKHIZK5SEVLSPRLKOWQT2R36MBUE3IDTOAKLZRNOA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.921054485 (632h53m14.79213752s) [noise=true , address=145.79.227.5:4221 , piece_id=SWHMPK7JYIVDPJJYXIM5YS5YGD3QNOF5GO57NK3TA226YPUWBEWA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.921424640 (632h53m14.792507686s) [noise=true , address=145.79.227.52:4445 , piece_id=TYB3H7ZEA5WVDT7F2REXCGEWYBF5TKL7PP5RQLWBHIDQWTFWOBVQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.923275004 (632h53m14.794358039s) [noise=true , address=145.79.227.4:4186 , piece_id=QCMIEG2NDHYFCXIR3CASGUEJEXKRDDQPU6X46E3YDOSNEWGWP7LQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.934454711 (632h53m14.805537746s) [noise=true , address=145.79.227.18:4007 , piece_id=TPBTA7BC332TCQVFWGEHW5IIUPX7VWVUIXLNEU6Y2GYR6UUV27TQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.946354569 (632h53m14.817437604s) [noise=true , address=145.79.227.27:4690 , piece_id=FB64RYKQ6CXCF2L6A55XIRUBZJRQV3VQZOSK4WAWN77CKFVFLJ3Q, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.947893469 (632h53m14.818976504s) [noise=true , address=145.79.227.26:4655 , piece_id=GMUHN5KKFONXNLMOAI5BCQJ36MR6RMSHIGVDPW4RUVZICX6T32LQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.951993707 (632h53m14.823076752s) [noise=true , address=145.79.227.50:4375 , piece_id=DQLX7VHWDBOFHXAJSIO5YDVUMWA6CGMI7ZRVV3AD3UTJWROH4FOQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:14.983034743 (632h53m14.854117788s) [noise=true , address=145.79.227.16:4912 , piece_id=XX7ZEGC4ZZGKD3YVQYC2BSKEYJKFGGNZIF63SL4ARF3XTCUJPWMQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.203948107 (632h53m15.075031152s) [noise=true , address=45.91.116.7:4010 , piece_id=MHPDUBDLGIV5GRETMRGFDKD52KWSRD4AHMQCIJQ44GMAIEGOYCKA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.216064908 (632h53m15.087147944s) [noise=true , address=77.90.31.17:4773 , piece_id=JYUVRRIMAU2OSSH5LCRRMUHHUJAFXZE3VYK5QUO4YBUXKV46ODQQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.216273646 (632h53m15.087356691s) [noise=true , address=77.90.31.10:4520 , piece_id=WJUSK2HNXXWXO2ZOFEP3J5AOOXFLKOL6IVZQFHWTRKT6A6GL5ZNA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.342047284 (632h53m15.213130319s) [noise=true , address=45.91.116.8:4046 , piece_id=RUQA7G3YWAIKSYCDZKEO2U2ZHMDDUP4HODXJ3ZNIJHHF3ZHMJ7AQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: <nil> Jan 1 00:22:15.342084845 (632h53m15.21316789s) [seg_pos={0 0}, upload=554577]: Segment reached optimal threshold of 13 pieces. Jan 1 00:22:15.342300776 (632h53m15.213383821s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Jan 1 00:22:15.342308010 (632h53m15.213391045s) [noise=true , address=45.91.116.4:4937 , piece_id=SOK2GKG6UUMOCUFRBKXPSQJL2ER74RMVQSYIQMQO5TLZZ6AWRNPQ, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342356141 (632h53m15.213439186s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 1 00:22:15.342372963 (632h53m15.213456008s) [noise=true , address=45.91.116.14:4192 , piece_id=HTQVA4QSOWTE3N7ID2WVQUHOPKF6N2ZPB3INBU7J6EASWLWQOOUA, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342726086 (632h53m15.213809121s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33692->77.90.31.16:4735: use of closed network connection Jan 1 00:22:15.342735083 (632h53m15.213818128s) [noise=true , address=77.90.31.16:4735 , piece_id=HEXGFPUK7TG5UJ7P2KKNFCFJ7KRP7YOW3CFW5TR5SM6BOJF5DZUA, seg_pos={0 0}, upload=554577]: Long tail cancellation due to optimal threshold Jan 1 00:22:15.342757906 (632h53m15.213840941s) [seg_pos={0 0}, upload=554577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:22:15.342836896 (632h53m15.213919932s) [upload=554577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:22:15.342853028 (632h53m15.213936063s) [upload=554577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:22:15.639323159 (632h53m15.510406195s) [upload=554577]: Done uploading segments... Jan 1 00:22:15.639343898 (632h53m15.510426933s) [upload=554577]: Upload finished. err: <nil> Duration: 30.003251909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X....................X...........X..X...........] duration: 26.484280114s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:22:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 26.484296265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X....................X...........X..X..........] duration: 35.44299039s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:22:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 35.443086373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......X....X....X......X....X....X.....XXX..X..XX.......XXX] duration: 30.002692058s trace: https://jaeger.dev.storj.io/trace/1f36590ce20e0ec2 FAIL (2026-01-01 00:22:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: DT3N4LQ7XW2CBIEEHLRAEW67QSRO3PKNZWD6U3CHGBRMRHHGCWNA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: LWQXDNWKTTRCN6V5GCQS56SMHVRUL42MEAI2LVFHCFSPIQA5OX5A) context deadline exceeded ======LOGS====== 1 00:21:33.630848542 (632h52m33.501931577s) [upload=554565]: Starting upload Jan 1 00:21:33.630946107 (632h52m33.502029172s) [upload=554565]: Done starting upload Jan 1 00:21:33.631043953 (632h52m33.502127018s) [upload=554565]: Uploading segments... Jan 1 00:21:33.631359364 (632h52m33.502442409s) [upload=554565]: Got next segment. Inline: false Jan 1 00:21:33.631405412 (632h52m33.502488457s) [upload=554565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:21:33.631411503 (632h52m33.502494548s) [upload=554565]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:21:33.876722262 (632h52m33.747805297s) [seg_pos={0 0}, upload=554565]: Begin upload segment... Jan 1 00:21:33.876763361 (632h52m33.747846396s) [seg_pos={0 0}, upload=554565]: Done begin upload segment. Jan 1 00:21:33.876788228 (632h52m33.747871263s) [noise=true , address=45.91.116.5:4300 , piece_id=DT3N4LQ7XW2CBIEEHLRAEW67QSRO3PKNZWD6U3CHGBRMRHHGCWNA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.876816542 (632h52m33.747899587s) [noise=true , address=145.79.227.19:4078 , piece_id=ERK3BZUBUPL6BJI6VOG7ANMYCA6M7Q3F4XSPHC4RQJXWBINGE5EA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.877252221 (632h52m33.748335256s) [noise=true , address=145.79.227.48:4301 , piece_id=A4ITT5G2DMH7JTRCHBB6OL3TGIEFYD4D2UGGAO6Q5RUXQTL3KQOA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.877634249 (632h52m33.748717284s) [noise=true , address=45.91.116.10:4083 , piece_id=D3O6AMCG5ADSCKZQH3Y47VHURMVAVADMSUV27YMQAS3HQJ2WLLKA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.877969017 (632h52m33.749052062s) [noise=true , address=145.79.227.17:4971 , piece_id=XNMYSD3JSWNRNU76C5LL2IL56D7T5KMMZIACPYCYU6NORNKF67TQ, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.878337498 (632h52m33.749420543s) [noise=true , address=45.91.116.6:4973 , piece_id=WD73R5DNESSGFWC6DPSXWNFW3U7NYXI4D3IGKYKJPQXK2RGHIBGQ, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.878692154 (632h52m33.749775189s) [noise=true , address=77.90.31.13:4626 , piece_id=L5XW3F52EPWNKFZ4CJ4ZE5SG6WPKCANL7B2FN6P5KM2M55BFMF4A, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.879025518 (632h52m33.750108563s) [noise=true , address=77.90.31.17:4770 , piece_id=P5OJ2TCKQ2P2XGLQFLO7YOE3VX6NTRF6ZF74SF2ODOY5OKPLYINA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.879372829 (632h52m33.750455875s) [noise=true , address=145.79.227.46:4602 , piece_id=74GPSADMNEKE73SYOEMHH274OF7YUEKJQVUD3SUDK5CI2HQRFE5A, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.880784026 (632h52m33.751867061s) [noise=true , address=77.90.31.19:4842 , piece_id=LWQXDNWKTTRCN6V5GCQS56SMHVRUL42MEAI2LVFHCFSPIQA5OX5A, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.882003869 (632h52m33.753086904s) [noise=true , address=145.79.227.51:4411 , piece_id=2XEHASGSUE7I7IUWOAAWTNZPBVMTNCAJ33U6F5FAHH6HTGGH52VA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.883363027 (632h52m33.754446072s) [noise=true , address=145.79.227.14:4837 , piece_id=W3EUA3WO2AXNHZV5IR3OCBCL772HBJR34WMXGKEHJCMDP2JOVJDA, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.884625782 (632h52m33.755708817s) [noise=true , address=145.79.227.5:4222 , piece_id=T3IWYN2FDODLRIM7JO6WG6FRQC56GHBZPZEG6HBIHWL4N7LA4WOQ, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.886283528 (632h52m33.757366563s) [noise=true , address=145.79.227.20:4438 , piece_id=5UP3CT4HUIF5E7AN7KIBTRUUQZOZIHK77V3FNG74PFMHTCRAIROQ, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.889438566 (632h52m33.760521601s) [noise=true , address=77.90.31.10:4519 , piece_id=I6TJDIULSJBROSYO4LORISMMLUMWTC4ZR3ZAZTRVHP3MEHKAJU7Q, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:33.890686111 (632h52m33.761769156s) [noise=true , address=45.91.116.9:4263 , piece_id=DB4JDIWLIICNOOFPTEEB6Q2WOHETH47DZU4E6ZCXGWJO6KBZ7ULQ, seg_pos={0 0}, upload=554565]: Uploading piece... Jan 1 00:21:35.590675361 (632h52m35.461758396s) [upload=554565]: Next returned nil segment Jan 1 00:21:35.590700298 (632h52m35.461783363s) [upload=554565]: Waiting for error group managing segments... Jan 1 00:21:35.874771532 (632h52m35.745854567s) [noise=true , address=145.79.227.5:4222 , piece_id=T3IWYN2FDODLRIM7JO6WG6FRQC56GHBZPZEG6HBIHWL4N7LA4WOQ, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.881318347 (632h52m35.752401382s) [noise=true , address=145.79.227.46:4602 , piece_id=74GPSADMNEKE73SYOEMHH274OF7YUEKJQVUD3SUDK5CI2HQRFE5A, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.881965350 (632h52m35.753048385s) [noise=true , address=145.79.227.51:4411 , piece_id=2XEHASGSUE7I7IUWOAAWTNZPBVMTNCAJ33U6F5FAHH6HTGGH52VA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.886008498 (632h52m35.757091543s) [noise=true , address=145.79.227.19:4078 , piece_id=ERK3BZUBUPL6BJI6VOG7ANMYCA6M7Q3F4XSPHC4RQJXWBINGE5EA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.890818297 (632h52m35.761901352s) [noise=true , address=145.79.227.48:4301 , piece_id=A4ITT5G2DMH7JTRCHBB6OL3TGIEFYD4D2UGGAO6Q5RUXQTL3KQOA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.895023022 (632h52m35.766106068s) [noise=true , address=145.79.227.20:4438 , piece_id=5UP3CT4HUIF5E7AN7KIBTRUUQZOZIHK77V3FNG74PFMHTCRAIROQ, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:35.901113640 (632h52m35.772196685s) [noise=true , address=145.79.227.14:4837 , piece_id=W3EUA3WO2AXNHZV5IR3OCBCL772HBJR34WMXGKEHJCMDP2JOVJDA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.032899505 (632h52m35.90398255s) [noise=true , address=77.90.31.13:4626 , piece_id=L5XW3F52EPWNKFZ4CJ4ZE5SG6WPKCANL7B2FN6P5KM2M55BFMF4A, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.113508915 (632h52m35.98459195s) [noise=true , address=77.90.31.10:4519 , piece_id=I6TJDIULSJBROSYO4LORISMMLUMWTC4ZR3ZAZTRVHP3MEHKAJU7Q, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.156699655 (632h52m36.02778269s) [noise=true , address=145.79.227.17:4971 , piece_id=XNMYSD3JSWNRNU76C5LL2IL56D7T5KMMZIACPYCYU6NORNKF67TQ, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.192463022 (632h52m36.063546067s) [noise=true , address=45.91.116.6:4973 , piece_id=WD73R5DNESSGFWC6DPSXWNFW3U7NYXI4D3IGKYKJPQXK2RGHIBGQ, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.284587938 (632h52m36.155670974s) [noise=true , address=45.91.116.5:4300 , piece_id=DT3N4LQ7XW2CBIEEHLRAEW67QSRO3PKNZWD6U3CHGBRMRHHGCWNA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.820064066 (632h52m36.691147101s) [noise=true , address=77.90.31.19:4842 , piece_id=LWQXDNWKTTRCN6V5GCQS56SMHVRUL42MEAI2LVFHCFSPIQA5OX5A, seg_pos={0 0}, upload=554565]: Done uploading piece. err: <nil> Jan 1 00:21:36.820127145 (632h52m36.69121019s) [seg_pos={0 0}, upload=554565]: Segment reached optimal threshold of 13 pieces. Jan 1 00:21:36.820294364 (632h52m36.691377399s) [noise=true , address=45.91.116.9:4263 , piece_id=DB4JDIWLIICNOOFPTEEB6Q2WOHETH47DZU4E6ZCXGWJO6KBZ7ULQ, seg_pos={0 0}, upload=554565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:45112->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:45112->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45112->45.91.116.9:4263: use of closed network connection Jan 1 00:21:36.820300636 (632h52m36.691383671s) [noise=true , address=45.91.116.9:4263 , piece_id=DB4JDIWLIICNOOFPTEEB6Q2WOHETH47DZU4E6ZCXGWJO6KBZ7ULQ, seg_pos={0 0}, upload=554565]: Long tail cancellation due to optimal threshold Jan 1 00:21:36.820374376 (632h52m36.691457421s) [noise=true , address=77.90.31.17:4770 , piece_id=P5OJ2TCKQ2P2XGLQFLO7YOE3VX6NTRF6ZF74SF2ODOY5OKPLYINA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jan 1 00:21:36.820404023 (632h52m36.691487058s) [noise=true , address=77.90.31.17:4770 , piece_id=P5OJ2TCKQ2P2XGLQFLO7YOE3VX6NTRF6ZF74SF2ODOY5OKPLYINA, seg_pos={0 0}, upload=554565]: Long tail cancellation due to optimal threshold Jan 1 00:21:36.820519924 (632h52m36.691602959s) [noise=true , address=45.91.116.10:4083 , piece_id=D3O6AMCG5ADSCKZQH3Y47VHURMVAVADMSUV27YMQAS3HQJ2WLLKA, seg_pos={0 0}, upload=554565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 1 00:21:36.820526576 (632h52m36.691609611s) [noise=true , address=45.91.116.10:4083 , piece_id=D3O6AMCG5ADSCKZQH3Y47VHURMVAVADMSUV27YMQAS3HQJ2WLLKA, seg_pos={0 0}, upload=554565]: Long tail cancellation due to optimal threshold Jan 1 00:21:36.820542326 (632h52m36.691625371s) [seg_pos={0 0}, upload=554565]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:21:36.820607059 (632h52m36.691690094s) [upload=554565]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:21:36.820611828 (632h52m36.691694883s) [upload=554565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:21:37.129381360 (632h52m37.000464395s) [upload=554565]: Done uploading segments... Jan 1 00:21:37.129402531 (632h52m37.000485566s) [upload=554565]: Upload finished. err: <nil> Duration: 30.00271907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........X....X....X......X....X....X.....XXX..X..XX.......XX] duration: 30.003160152s trace: https://jaeger.dev.storj.io/trace/4f76ac31dd4a689a FAIL (2026-01-01 00:21:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 00:21:00.439449313 (632h52m0.310532348s) [upload=554559]: Starting upload Jan 1 00:21:00.439509809 (632h52m0.310592854s) [upload=554559]: Done starting upload Jan 1 00:21:00.439599529 (632h52m0.310682564s) [upload=554559]: Uploading segments... Jan 1 00:21:00.439697315 (632h52m0.31078035s) [upload=554559]: Got next segment. Inline: false Jan 1 00:21:00.439733825 (632h52m0.31081686s) [upload=554559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:21:00.439738293 (632h52m0.310821328s) [upload=554559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:21:00.689726850 (632h52m0.560809885s) [seg_pos={0 0}, upload=554559]: Begin upload segment... Jan 1 00:21:00.689775953 (632h52m0.560858988s) [seg_pos={0 0}, upload=554559]: Done begin upload segment. Jan 1 00:21:00.689836920 (632h52m0.560919955s) [noise=true , address=45.91.116.7:4012 , piece_id=2NRIZ6ZSRAHJTKFFRASP5NGJODACHBO77VZKYFLPQFWUBAVL3W2Q, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.689954383 (632h52m0.561037428s) [noise=true , address=145.79.227.18:4006 , piece_id=N6KNUCPTXLHDM4JRJH3M5NCMBCB3NBXUYLENFYGK3E6MJVWNCPWA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.690497097 (632h52m0.561580142s) [noise=true , address=145.79.227.10:4403 , piece_id=RFOMOAVHECHJRI7SACAD5T3MSKVTSKQIFIB67MREPM24GFYKBNOA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.691003051 (632h52m0.562086096s) [noise=true , address=145.79.227.46:4601 , piece_id=74ZPHMXL23MP7PLEYGXQEAYL6AOP7Q67YYCSFEYK2QXBRPQQZTYQ, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.691059177 (632h52m0.562142222s) [noise=true , address=145.79.227.27:4690 , piece_id=TX4NTRDLW4ZXVGXIU34MPRSX2AFMKI3JX3AFTGBUS6ZBF7MM2HXA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.691434041 (632h52m0.562517076s) [noise=true , address=77.90.31.15:4700 , piece_id=B7MZROSA6W65OITURSTCM7HYYX2A5T3AECTIQVQK5YQZ2FXD44YQ, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.691754000 (632h52m0.562837035s) [noise=true , address=145.79.227.7:4293 , piece_id=XL4BBMISRRZKIEHXKZNBIEPFNEB2ANDGJEF2HGDGSZTZAAKRAJCA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.691778888 (632h52m0.562861933s) [noise=true , address=145.79.227.12:4767 , piece_id=LPRJFKZM3G42VAKHJ6G7OA5WRASSKRNIWMQJE2EJSYT4OKOFLJEA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.692066946 (632h52m0.563149981s) [noise=true , address=145.79.227.17:4971 , piece_id=HSUKY3YJZVHD2STH7RZPDFTN5HFQAWFJNRWK3VJX3EN7NZZXN5XQ, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.692436500 (632h52m0.563519575s) [noise=true , address=77.90.31.17:4770 , piece_id=TF6UASJBBHPQLXZ5ULGLJ5E2BJKJY2U4MUVMB5XU3JJCXP5LZZ6Q, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.692458662 (632h52m0.563541697s) [noise=true , address=45.91.116.4:4940 , piece_id=ZCWQBZLIJ2OUH6QUI4MDLAIU4KJ2QAHHMGXLXTINLEAKVMTX6Q7A, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.692867280 (632h52m0.563950315s) [noise=true , address=77.90.31.13:4626 , piece_id=WFZIADBXP2YA5SSKOX4AL32AUS3W7SXUDROO62A4CKYXUE66HTYA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.693184053 (632h52m0.564267098s) [noise=true , address=145.79.227.41:4087 , piece_id=TZL2YWVZDPSJIWRORLH2NIIWGP4B7SWT75ETAOKWJIY7E6H2VXHQ, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.694008924 (632h52m0.565091959s) [noise=true , address=145.79.227.34:4905 , piece_id=TPSGJS73IS3JRHTL5KKIJKC5ULTY4Q47IKHMUICIJ44IPYTFVAQA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.694451718 (632h52m0.565534753s) [noise=true , address=145.79.227.5:4221 , piece_id=7J2ONDDC4MV25Y7MLWDNOSN3T25L5CNMY2C6UB6UQPN5CHXQCO6A, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:00.694778459 (632h52m0.565861494s) [noise=true , address=77.90.31.10:4520 , piece_id=RQWOIAD6MAKD3Y2FUSQYJX2BL2UAOYO4Z5HERZP6XX4WRWDBFMMA, seg_pos={0 0}, upload=554559]: Uploading piece... Jan 1 00:21:02.385856803 (632h52m2.256939848s) [upload=554559]: Next returned nil segment Jan 1 00:21:02.385880117 (632h52m2.256963162s) [upload=554559]: Waiting for error group managing segments... Jan 1 00:21:02.669350838 (632h52m2.540433873s) [noise=true , address=145.79.227.10:4403 , piece_id=RFOMOAVHECHJRI7SACAD5T3MSKVTSKQIFIB67MREPM24GFYKBNOA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.678588578 (632h52m2.549671623s) [noise=true , address=145.79.227.18:4006 , piece_id=N6KNUCPTXLHDM4JRJH3M5NCMBCB3NBXUYLENFYGK3E6MJVWNCPWA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.683946510 (632h52m2.555029555s) [noise=true , address=145.79.227.46:4601 , piece_id=74ZPHMXL23MP7PLEYGXQEAYL6AOP7Q67YYCSFEYK2QXBRPQQZTYQ, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.685121086 (632h52m2.556204131s) [noise=true , address=145.79.227.27:4690 , piece_id=TX4NTRDLW4ZXVGXIU34MPRSX2AFMKI3JX3AFTGBUS6ZBF7MM2HXA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.690715128 (632h52m2.561798163s) [noise=true , address=145.79.227.17:4971 , piece_id=HSUKY3YJZVHD2STH7RZPDFTN5HFQAWFJNRWK3VJX3EN7NZZXN5XQ, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.698269993 (632h52m2.569353039s) [noise=true , address=145.79.227.5:4221 , piece_id=7J2ONDDC4MV25Y7MLWDNOSN3T25L5CNMY2C6UB6UQPN5CHXQCO6A, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.702874671 (632h52m2.573957716s) [noise=true , address=145.79.227.41:4087 , piece_id=TZL2YWVZDPSJIWRORLH2NIIWGP4B7SWT75ETAOKWJIY7E6H2VXHQ, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.711320864 (632h52m2.582403899s) [noise=true , address=145.79.227.12:4767 , piece_id=LPRJFKZM3G42VAKHJ6G7OA5WRASSKRNIWMQJE2EJSYT4OKOFLJEA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.846944697 (632h52m2.718027742s) [noise=true , address=77.90.31.13:4626 , piece_id=WFZIADBXP2YA5SSKOX4AL32AUS3W7SXUDROO62A4CKYXUE66HTYA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.967381441 (632h52m2.838464487s) [noise=true , address=77.90.31.10:4520 , piece_id=RQWOIAD6MAKD3Y2FUSQYJX2BL2UAOYO4Z5HERZP6XX4WRWDBFMMA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.977755275 (632h52m2.84883831s) [noise=true , address=45.91.116.4:4940 , piece_id=ZCWQBZLIJ2OUH6QUI4MDLAIU4KJ2QAHHMGXLXTINLEAKVMTX6Q7A, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:02.989714928 (632h52m2.860797973s) [noise=true , address=45.91.116.7:4012 , piece_id=2NRIZ6ZSRAHJTKFFRASP5NGJODACHBO77VZKYFLPQFWUBAVL3W2Q, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:03.573856028 (632h52m3.444939073s) [noise=true , address=77.90.31.17:4770 , piece_id=TF6UASJBBHPQLXZ5ULGLJ5E2BJKJY2U4MUVMB5XU3JJCXP5LZZ6Q, seg_pos={0 0}, upload=554559]: Done uploading piece. err: <nil> Jan 1 00:21:03.573885193 (632h52m3.444968239s) [seg_pos={0 0}, upload=554559]: Segment reached optimal threshold of 13 pieces. Jan 1 00:21:03.574217155 (632h52m3.44530021s) [noise=true , address=145.79.227.7:4293 , piece_id=XL4BBMISRRZKIEHXKZNBIEPFNEB2ANDGJEF2HGDGSZTZAAKRAJCA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:40334->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:40334->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40334->145.79.227.7:4293: use of closed network connection Jan 1 00:21:03.574228127 (632h52m3.445311172s) [noise=true , address=145.79.227.7:4293 , piece_id=XL4BBMISRRZKIEHXKZNBIEPFNEB2ANDGJEF2HGDGSZTZAAKRAJCA, seg_pos={0 0}, upload=554559]: Long tail cancellation due to optimal threshold Jan 1 00:21:03.574233948 (632h52m3.445316993s) [noise=true , address=77.90.31.15:4700 , piece_id=B7MZROSA6W65OITURSTCM7HYYX2A5T3AECTIQVQK5YQZ2FXD44YQ, seg_pos={0 0}, upload=554559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Jan 1 00:21:03.574250659 (632h52m3.445333694s) [noise=true , address=77.90.31.15:4700 , piece_id=B7MZROSA6W65OITURSTCM7HYYX2A5T3AECTIQVQK5YQZ2FXD44YQ, seg_pos={0 0}, upload=554559]: Long tail cancellation due to optimal threshold Jan 1 00:21:03.574489824 (632h52m3.445572869s) [noise=true , address=145.79.227.34:4905 , piece_id=TPSGJS73IS3JRHTL5KKIJKC5ULTY4Q47IKHMUICIJ44IPYTFVAQA, seg_pos={0 0}, upload=554559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 1 00:21:03.574497619 (632h52m3.445580664s) [noise=true , address=145.79.227.34:4905 , piece_id=TPSGJS73IS3JRHTL5KKIJKC5ULTY4Q47IKHMUICIJ44IPYTFVAQA, seg_pos={0 0}, upload=554559]: Long tail cancellation due to optimal threshold Jan 1 00:21:03.574684335 (632h52m3.44576738s) [seg_pos={0 0}, upload=554559]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:21:03.574809633 (632h52m3.445892678s) [upload=554559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:21:03.574817318 (632h52m3.445900363s) [upload=554559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:21:03.875621995 (632h52m3.74670504s) [upload=554559]: Done uploading segments... Jan 1 00:21:03.875643725 (632h52m3.74672677s) [upload=554559]: Upload finished. err: <nil> Duration: 30.00318086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X....................X...........X..X.........] duration: 21.446036821s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:21:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 21.446053715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X....................X...........X..X........] duration: 32.166741072s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:20:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 32.166771271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........X....X....X......X....X....X.....XXX..X..XX.......X] duration: 30.002553978s trace: https://jaeger.dev.storj.io/trace/42734d11284293f4 FAIL (2026-01-01 00:20:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: AJ4JM3GIVUV6SN6H7DO57EWC3DNUMSJZVXLHPWU2VLVMISZMIHBA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: QULGJQGLCDRPT3PHD3IYNWMD5EVIUJN7NCGYQK2KPDZEEGPMXRAA) context deadline exceeded ======LOGS====== 0734s) [upload=554545]: Uploading segments... Jan 1 00:20:07.677148691 (632h51m7.548231726s) [upload=554545]: Got next segment. Inline: false Jan 1 00:20:07.677198635 (632h51m7.54828167s) [upload=554545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 00:20:07.677207251 (632h51m7.548290327s) [upload=554545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 00:20:07.926271698 (632h51m7.797354743s) [seg_pos={0 0}, upload=554545]: Begin upload segment... Jan 1 00:20:07.926347072 (632h51m7.797430117s) [seg_pos={0 0}, upload=554545]: Done begin upload segment. Jan 1 00:20:07.926397147 (632h51m7.797480192s) [noise=true , address=145.79.227.8:4329 , piece_id=FGAFTS5L3WCYHM4C6MDBR6ECS4Z3MIYIMALX6ZPDJJ3XSGQEEEPQ, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.926404420 (632h51m7.797487455s) [noise=true , address=145.79.227.50:4374 , piece_id=UTLQPODPZVEBH7KTKIB5PJVCMKLVGKMNRDBKJVBUDUBVMOYUAH5Q, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.927633431 (632h51m7.798716476s) [noise=true , address=45.91.116.10:4084 , piece_id=7HDGWCORGC2IRAHUDQBBG4R5KEYWYHTXSJV4GZSWZT7HDOZDYTFQ, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.928195832 (632h51m7.799278877s) [noise=true , address=145.79.227.41:4087 , piece_id=OGNUJMTC2MXSCAWU3SMULUWKBGRGFNBZBNGTPPH34DYMG6O6H3FA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.928698709 (632h51m7.799781754s) [noise=true , address=145.79.227.10:4403 , piece_id=OTME4XMJCXS4YI6DEBZEEFD76VAG5BGYB4FSBGAGFW4PYZD72QQA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.929177762 (632h51m7.800260797s) [noise=true , address=45.91.116.13:4153 , piece_id=DDNJLAZDZXG4BCG5YAEIUSYNV5XXKYGLJXUQTEDOOEJ2FRNRYWKA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.929188872 (632h51m7.800271917s) [noise=true , address=145.79.227.42:4121 , piece_id=VBZRP5HSKD3ILSGOQXLJTVMMAJFZSWSKDDYZXLKFPTI3J3BQ5RMA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.929639130 (632h51m7.800722175s) [noise=true , address=145.79.227.12:4765 , piece_id=IV7TL4SHQIVM6VXMCIELJ7ALODEY2YTW4JTW7PQRKWOD27ARDZZA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.929794837 (632h51m7.800877882s) [noise=true , address=45.91.116.5:4298 , piece_id=T6LLFL2PYC6VWFKFKRBEDGJZ2EAHPQ3B4RIFVUF3GJETYEJENEOA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.930213534 (632h51m7.801296579s) [noise=true , address=77.90.31.13:4627 , piece_id=AJ4JM3GIVUV6SN6H7DO57EWC3DNUMSJZVXLHPWU2VLVMISZMIHBA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.930740517 (632h51m7.801823552s) [noise=true , address=77.90.31.11:4554 , piece_id=KGMH7SSYWOZLI2ELTSQFYIS2SDJMLKDNVH4AFR5DCQLBMMEDTB6A, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.930875825 (632h51m7.80195887s) [noise=true , address=45.91.116.4:4939 , piece_id=LQ7XGBX6KKIKDHQZHVTKP7CKUY3AYJFMG7UMEVSRSTHK3DGSFM2Q, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.931250198 (632h51m7.802333253s) [noise=true , address=145.79.227.4:4186 , piece_id=JYN7LDBIMYGUXWALY7VTABQI6ZZ5GQXGJ4DHDZIR7XN3Y2HYN6KQ, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.931784235 (632h51m7.80286727s) [noise=true , address=45.91.116.11:4119 , piece_id=XIU2UAKGRIIK7CW7LLNWKXZFHY3SGFZ7DZ7BY4QVS7KP64UK5FNQ, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.931998944 (632h51m7.803081989s) [noise=true , address=77.90.31.21:4877 , piece_id=QULGJQGLCDRPT3PHD3IYNWMD5EVIUJN7NCGYQK2KPDZEEGPMXRAA, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:07.932303103 (632h51m7.803386158s) [noise=true , address=145.79.227.11:4732 , piece_id=QGVVSC6DU3MYZYU3HC6ZBG4YQDND7PISGNDC7PMQRKFTW6FOBZ4Q, seg_pos={0 0}, upload=554545]: Uploading piece... Jan 1 00:20:09.640073018 (632h51m9.511156053s) [upload=554545]: Next returned nil segment Jan 1 00:20:09.640091534 (632h51m9.511174579s) [upload=554545]: Waiting for error group managing segments... Jan 1 00:20:09.929463480 (632h51m9.800546525s) [noise=true , address=145.79.227.50:4374 , piece_id=UTLQPODPZVEBH7KTKIB5PJVCMKLVGKMNRDBKJVBUDUBVMOYUAH5Q, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:09.935030982 (632h51m9.806114027s) [noise=true , address=145.79.227.41:4087 , piece_id=OGNUJMTC2MXSCAWU3SMULUWKBGRGFNBZBNGTPPH34DYMG6O6H3FA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:09.942689664 (632h51m9.813772709s) [noise=true , address=145.79.227.4:4186 , piece_id=JYN7LDBIMYGUXWALY7VTABQI6ZZ5GQXGJ4DHDZIR7XN3Y2HYN6KQ, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:09.944484212 (632h51m9.815567257s) [noise=true , address=145.79.227.11:4732 , piece_id=QGVVSC6DU3MYZYU3HC6ZBG4YQDND7PISGNDC7PMQRKFTW6FOBZ4Q, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:09.944720061 (632h51m9.815803106s) [noise=true , address=145.79.227.8:4329 , piece_id=FGAFTS5L3WCYHM4C6MDBR6ECS4Z3MIYIMALX6ZPDJJ3XSGQEEEPQ, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:09.980720801 (632h51m9.851803846s) [noise=true , address=145.79.227.12:4765 , piece_id=IV7TL4SHQIVM6VXMCIELJ7ALODEY2YTW4JTW7PQRKWOD27ARDZZA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.001444972 (632h51m9.872528027s) [noise=true , address=145.79.227.42:4121 , piece_id=VBZRP5HSKD3ILSGOQXLJTVMMAJFZSWSKDDYZXLKFPTI3J3BQ5RMA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.006359599 (632h51m9.877442634s) [noise=true , address=145.79.227.10:4403 , piece_id=OTME4XMJCXS4YI6DEBZEEFD76VAG5BGYB4FSBGAGFW4PYZD72QQA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.120073393 (632h51m9.991156428s) [noise=true , address=77.90.31.13:4627 , piece_id=AJ4JM3GIVUV6SN6H7DO57EWC3DNUMSJZVXLHPWU2VLVMISZMIHBA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.212869397 (632h51m10.083952462s) [noise=true , address=45.91.116.11:4119 , piece_id=XIU2UAKGRIIK7CW7LLNWKXZFHY3SGFZ7DZ7BY4QVS7KP64UK5FNQ, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.221826693 (632h51m10.092909738s) [noise=true , address=77.90.31.21:4877 , piece_id=QULGJQGLCDRPT3PHD3IYNWMD5EVIUJN7NCGYQK2KPDZEEGPMXRAA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.264804738 (632h51m10.135887783s) [noise=true , address=45.91.116.5:4298 , piece_id=T6LLFL2PYC6VWFKFKRBEDGJZ2EAHPQ3B4RIFVUF3GJETYEJENEOA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.344870752 (632h51m10.215953797s) [noise=true , address=45.91.116.10:4084 , piece_id=7HDGWCORGC2IRAHUDQBBG4R5KEYWYHTXSJV4GZSWZT7HDOZDYTFQ, seg_pos={0 0}, upload=554545]: Done uploading piece. err: <nil> Jan 1 00:20:10.344894267 (632h51m10.215977312s) [seg_pos={0 0}, upload=554545]: Segment reached optimal threshold of 13 pieces. Jan 1 00:20:10.345081023 (632h51m10.216164058s) [noise=true , address=77.90.31.11:4554 , piece_id=KGMH7SSYWOZLI2ELTSQFYIS2SDJMLKDNVH4AFR5DCQLBMMEDTB6A, seg_pos={0 0}, upload=554545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:60870->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:60870->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->77.90.31.11:4554: use of closed network connection Jan 1 00:20:10.345087585 (632h51m10.21617063s) [noise=true , address=77.90.31.11:4554 , piece_id=KGMH7SSYWOZLI2ELTSQFYIS2SDJMLKDNVH4AFR5DCQLBMMEDTB6A, seg_pos={0 0}, upload=554545]: Long tail cancellation due to optimal threshold Jan 1 00:20:10.345149904 (632h51m10.216232949s) [noise=true , address=45.91.116.4:4939 , piece_id=LQ7XGBX6KKIKDHQZHVTKP7CKUY3AYJFMG7UMEVSRSTHK3DGSFM2Q, seg_pos={0 0}, upload=554545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:33590->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:33590->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33590->45.91.116.4:4939: use of closed network connection Jan 1 00:20:10.345159221 (632h51m10.216242256s) [noise=true , address=45.91.116.4:4939 , piece_id=LQ7XGBX6KKIKDHQZHVTKP7CKUY3AYJFMG7UMEVSRSTHK3DGSFM2Q, seg_pos={0 0}, upload=554545]: Long tail cancellation due to optimal threshold Jan 1 00:20:10.345325428 (632h51m10.216408473s) [noise=true , address=45.91.116.13:4153 , piece_id=DDNJLAZDZXG4BCG5YAEIUSYNV5XXKYGLJXUQTEDOOEJ2FRNRYWKA, seg_pos={0 0}, upload=554545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Jan 1 00:20:10.345335156 (632h51m10.216418201s) [noise=true , address=45.91.116.13:4153 , piece_id=DDNJLAZDZXG4BCG5YAEIUSYNV5XXKYGLJXUQTEDOOEJ2FRNRYWKA, seg_pos={0 0}, upload=554545]: Long tail cancellation due to optimal threshold Jan 1 00:20:10.345356627 (632h51m10.216439663s) [seg_pos={0 0}, upload=554545]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 00:20:10.345448493 (632h51m10.216531528s) [upload=554545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:20:10.345454825 (632h51m10.21653787s) [upload=554545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:20:10.642646802 (632h51m10.513729837s) [upload=554545]: Done uploading segments... Jan 1 00:20:10.642676177 (632h51m10.513759222s) [upload=554545]: Upload finished. err: <nil> Duration: 30.00257517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X....................X...........X..X.......] duration: 15.302646472s trace: https://jaeger.dev.storj.io/trace/2fc75fe102d01309 FAIL (2026-01-01 00:19:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242031295 (632h44m26.11311433s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242171131 (632h44m26.113254166s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242287734 (632h44m26.113370769s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242468258 (632h44m26.113551293s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242616830 (632h44m26.113699875s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242884911 (632h44m26.113967946s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.242900751 (632h44m26.113983786s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243168461 (632h44m26.114251496s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243256328 (632h44m26.114339363s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243432653 (632h44m26.114515688s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243693821 (632h44m26.114776856s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.243725872 (632h44m26.114808907s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:26.244100656 (632h44m26.115183691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Uploading piece... Jan 1 00:13:27.652082260 (632h44m27.523165306s) [upload=554449]: Next returned nil segment Jan 1 00:13:27.652094083 (632h44m27.523177118s) [upload=554449]: Waiting for error group managing segments... Jan 1 00:13:28.012716308 (632h44m27.883799353s) [noise=true , address=64.49.13.154:4927 , piece_id=STHN2CPCYAOTT5JZPGDUWS7NUC5HBHZRIW4WZZFDAOY273WI75FQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.114071681 (632h44m27.985154716s) [noise=true , address=145.79.227.14:4859 , piece_id=5WJEROUVDHQGRYW4B6MW2WV2XFT2ANZJPC4HKQMAKBRLXOHIIHEQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.164070293 (632h44m28.035153328s) [noise=true , address=145.79.227.29:4963 , piece_id=MSDU27I2TFGOTK45A7M7RUFDN3DNMVV2OQSGFFTLFPKDGFPOYKNQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.263534474 (632h44m28.134617509s) [noise=true , address=77.90.31.15:4732 , piece_id=NNRY6NGBLZOL3N5YGIG7ODVPRVF5O4TC6Z42BQMK62UQW7LQLGQA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.342109691 (632h44m28.213192726s) [noise=true , address=5.231.73.2:4032 , piece_id=SVUKHQKOZR5RLP27277JRVENAAVHDFWGDSHY2ZRB65MDRT64UA4A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.409939456 (632h44m28.281022501s) [noise=true , address=77.90.31.13:4653 , piece_id=JVZSKNPBA34Z35BHXJYYLEUR3BOPKBO5HROX3FGBRQHSUIPV2KRQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:28.988476164 (632h44m28.859559199s) [noise=true , address=145.79.227.35:4487 , piece_id=4EGIIUT5ZZUGJ36QCP6KKK3C43MM5GP2XLNPWV5PQDTN2QSNTPYQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.004716147 (632h44m28.875799182s) [noise=true , address=145.79.227.40:4076 , piece_id=T6PGKRSXQP2S5ZLWUNDCDW2KLS5FYTOS2GKLN2F2MEXFDUQIKWZA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.008155126 (632h44m28.879238161s) [noise=true , address=145.79.227.49:4372 , piece_id=PXH4MBJCV5TYMI4MNZRFPIBROQ7SJHAAPIEW3S3X3X5TGOOH5FWA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.035414601 (632h44m28.906497636s) [noise=true , address=145.79.227.34:4923 , piece_id=IC3SUVWRCGPTHHVWX5NESGRTGS7SM77YALURA5YYOCI6DD4SJRMQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.068226467 (632h44m28.939309503s) [noise=true , address=145.79.227.30:4781 , piece_id=VLQTFGC2YAW4PWQE4VPYVQNYVZR4EB4MOEG33T3VRZ7KEA2GFV3A, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.114835698 (632h44m28.985918743s) [noise=true , address=145.79.227.43:4163 , piece_id=5FOATGRK35VYZ4JYOSFMHKR6JSKYR6OW2L4CNBGCRWUOMIBFRONA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.188415161 (632h44m29.059498196s) [noise=true , address=145.79.227.33:4893 , piece_id=JDK6ZWPWCXDVYRCESWOOGWFJN2FILUP6RKW2ZEEYE67OQX5DOUYA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307846712 (632h44m29.178929747s) [noise=true , address=77.90.31.21:4910 , piece_id=66C6HYSPSNLDMNIPW5FQTLW2ZMKHTIGXWFMUSSFF7DEUUGI4C5AQ, seg_pos={0 0}, upload=554449]: Done uploading piece. err: <nil> Jan 1 00:13:29.307870307 (632h44m29.178953342s) [seg_pos={0 0}, upload=554449]: Segment reached optimal threshold of 14 pieces. Jan 1 00:13:29.308120203 (632h44m29.179203248s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->145.79.227.18:4032: use of closed network connection Jan 1 00:13:29.308134861 (632h44m29.179217896s) [noise=true , address=145.79.227.18:4032 , piece_id=ED4OIK7334W3EYKHUKTTZUARWTRMELHEHL47NJSEPVTUIY2FYYXA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308347375 (632h44m29.17943042s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.50:4381: use of closed network connection Jan 1 00:13:29.308358656 (632h44m29.179441691s) [noise=true , address=145.79.227.50:4381 , piece_id=S6WUG7TV4AXFZAZDFMSGY7HKI4YFFKIEYLLJ436Z3OTZO7XJGU7Q, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308437597 (632h44m29.179520643s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17YmVUymv5Cx9wFw1g67CfEHxg8B75TcFkTCRpNdTm2PHanHVJ): protocol: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48476->145.79.227.16:4935: use of closed network connection Jan 1 00:13:29.308447997 (632h44m29.179531042s) [noise=true , address=145.79.227.16:4935 , piece_id=DNJ6CXXYNAMM5K5KVDRAS3WWFXY22OCHAM3JDQBKXACSRM2UAEHA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308532968 (632h44m29.179616003s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ex3KjvY38tkSstUAgALSN9i4tBKj2D24BfA6jWi34ntvqmJNT): protocol: expected piece hash; context canceled; EOF Jan 1 00:13:29.308542657 (632h44m29.179625693s) [noise=true , address=145.79.227.50:4390 , piece_id=LSXC4P3OOWX46R5WDPPXRO4VV3Y2DQWH3GDGN7GQBNKZCWSWHYKA, seg_pos={0 0}, upload=554449]: Long tail cancellation due to optimal threshold Jan 1 00:13:29.308563897 (632h44m29.179646942s) [seg_pos={0 0}, upload=554449]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 00:13:29.308646625 (632h44m29.179729671s) [upload=554449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 00:13:29.308652767 (632h44m29.179735812s) [upload=554449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 00:13:29.603620468 (632h44m29.474703503s) [upload=554449]: Done uploading segments... Jan 1 00:13:29.603639684 (632h44m29.474722719s) [upload=554449]: Upload finished. err: <nil> Duration: 15.302741964s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)