Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..X......XXXX.....XX.X.....X......X..............X....X.] duration: 6.026018069s trace: https://jaeger.dev.storj.io/trace/1365710d528f2261 FAIL (2025-12-28 09:22:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 09:21:45.602958640 (545h52m45.474041685s) [upload=478681]: Starting upload Dec 28 09:21:45.603049033 (545h52m45.474132078s) [upload=478681]: Done starting upload Dec 28 09:21:45.603105449 (545h52m45.474188495s) [upload=478681]: Uploading segments... Dec 28 09:21:45.603310941 (545h52m45.474393986s) [upload=478681]: Got next segment. Inline: false Dec 28 09:21:45.603350326 (545h52m45.474433361s) [upload=478681]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:21:45.603355666 (545h52m45.474438711s) [upload=478681]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:21:45.851113360 (545h52m45.722196405s) [seg_pos={0 0}, upload=478681]: Begin upload segment... Dec 28 09:21:45.851184555 (545h52m45.72226759s) [seg_pos={0 0}, upload=478681]: Done begin upload segment. Dec 28 09:21:45.851241674 (545h52m45.722324709s) [noise=true , address=145.79.227.11:4730 , piece_id=JA6CXABHZ5WWMV4RDP6KOV4UWA6VUDRB5QHFTFH7B5SF6WAXYHTA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.851250341 (545h52m45.722333406s) [noise=true , address=145.79.227.7:4295 , piece_id=JGHDNGCXX34W2EKWJ7NJWFNB6PSCMZLO25CUFSZ3AVT34ACLL3SQ, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.851949492 (545h52m45.723032587s) [noise=true , address=145.79.227.34:4905 , piece_id=OO5M4G7PB3QCX5HNXFATZQZPPFJ5D77VPTPPQQAP3HO4ZFSCS75A, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.852631200 (545h52m45.723714245s) [noise=true , address=145.79.227.6:4258 , piece_id=UTZ73VCSMKBBW5XKHTZVBIGTYU5Z4OEW6NX3NFTMK5BNTCO2ZZKQ, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.853444759 (545h52m45.724527804s) [noise=true , address=145.79.227.5:4221 , piece_id=WTLYKNU3AOEAQ6HBFVI44BXQG4J4LDCSPAXNM3PYE75GZL7PZMJA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.854012841 (545h52m45.725095886s) [noise=true , address=77.90.31.17:4770 , piece_id=NBCJK3Y4IN5RX6MX6T52MFQOQOGQV43CSHGGDXTOPNADU4UJKXHA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.854506450 (545h52m45.725589496s) [noise=true , address=145.79.227.39:4015 , piece_id=TLIY2RKO2NCJBZY7FVBZTDGU6UUW4SUPQSU5ZOJ4TMW5WKT4D23Q, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.855528596 (545h52m45.726611631s) [noise=true , address=77.90.31.11:4554 , piece_id=AT2VTWBHTAGU27QADGLEXISPWJCP2YHJTZ3AY4QB7XTRCFMMBL3A, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.856150761 (545h52m45.727233806s) [noise=true , address=145.79.227.16:4910 , piece_id=DP6DFFELJNGLE3TWKRHZL5HKCFKWJ4BQ5M5IT7MXGOYF5NPKTC6Q, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.857032962 (545h52m45.728115997s) [noise=true , address=45.91.116.12:4227 , piece_id=HFRQERH3CDDHZXGDZAVKICZKWQP2OWWK7NVUMC2ECNYEMMF4U53Q, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.858329170 (545h52m45.729412215s) [noise=true , address=77.90.31.13:4626 , piece_id=57EBKKWM5LSLSCDJTDYGCGLJHLEHUKAVQWOZSECM666SMRPJGLSA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.858476059 (545h52m45.729559104s) [noise=true , address=145.79.227.2:4113 , piece_id=YKRAXDV5JRXHB3RMLCSE62W4V6KTGCCS5I3SAIOJXCMPD64MFNCA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.859961638 (545h52m45.731044673s) [noise=true , address=145.79.227.14:4838 , piece_id=7TVKHNJANF66BA3Q3LHOK6LRRNIKR7YYGKSIXNSASMIMFNGXH3PA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.860087317 (545h52m45.731170352s) [noise=true , address=45.91.116.13:4153 , piece_id=KOACV2JXD72NTUFIRE6GXQRZLOPW4RWAOP5OFRK2FJKITMXMAKNQ, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.861968259 (545h52m45.733051314s) [noise=true , address=77.90.31.19:4844 , piece_id=OH34RGEN7WKPC5IYY3K7ULZH6GGXET3BG4C6SSWO2OX24OSGXRDA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.862533406 (545h52m45.733616441s) [noise=true , address=45.91.116.8:4047 , piece_id=SNFCR7XN3ZNVF6YSCPAZKP3AB2T3QKSEFS2WMOJXP5HN3GV2LSJA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:47.579080171 (545h52m47.450163206s) [upload=478681]: Next returned nil segment Dec 28 09:21:47.579092003 (545h52m47.450175038s) [upload=478681]: Waiting for error group managing segments... Dec 28 09:21:47.858894054 (545h52m47.729977099s) [noise=true , address=145.79.227.11:4730 , piece_id=JA6CXABHZ5WWMV4RDP6KOV4UWA6VUDRB5QHFTFH7B5SF6WAXYHTA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.860354184 (545h52m47.731437229s) [noise=true , address=145.79.227.5:4221 , piece_id=WTLYKNU3AOEAQ6HBFVI44BXQG4J4LDCSPAXNM3PYE75GZL7PZMJA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.862538885 (545h52m47.73362192s) [noise=true , address=145.79.227.2:4113 , piece_id=YKRAXDV5JRXHB3RMLCSE62W4V6KTGCCS5I3SAIOJXCMPD64MFNCA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.871318562 (545h52m47.742401597s) [noise=true , address=145.79.227.14:4838 , piece_id=7TVKHNJANF66BA3Q3LHOK6LRRNIKR7YYGKSIXNSASMIMFNGXH3PA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.882626314 (545h52m47.75370936s) [noise=true , address=145.79.227.7:4295 , piece_id=JGHDNGCXX34W2EKWJ7NJWFNB6PSCMZLO25CUFSZ3AVT34ACLL3SQ, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.884550719 (545h52m47.755633764s) [noise=true , address=145.79.227.16:4910 , piece_id=DP6DFFELJNGLE3TWKRHZL5HKCFKWJ4BQ5M5IT7MXGOYF5NPKTC6Q, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.889669025 (545h52m47.76075206s) [noise=true , address=145.79.227.6:4258 , piece_id=UTZ73VCSMKBBW5XKHTZVBIGTYU5Z4OEW6NX3NFTMK5BNTCO2ZZKQ, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.892803403 (545h52m47.763886438s) [noise=true , address=145.79.227.39:4015 , piece_id=TLIY2RKO2NCJBZY7FVBZTDGU6UUW4SUPQSU5ZOJ4TMW5WKT4D23Q, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.967012180 (545h52m47.838095225s) [noise=true , address=145.79.227.34:4905 , piece_id=OO5M4G7PB3QCX5HNXFATZQZPPFJ5D77VPTPPQQAP3HO4ZFSCS75A, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:48.090280493 (545h52m47.961363528s) [noise=true , address=45.91.116.8:4047 , piece_id=SNFCR7XN3ZNVF6YSCPAZKP3AB2T3QKSEFS2WMOJXP5HN3GV2LSJA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:48.117585636 (545h52m47.988668691s) [noise=true , address=45.91.116.12:4227 , piece_id=HFRQERH3CDDHZXGDZAVKICZKWQP2OWWK7NVUMC2ECNYEMMF4U53Q, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:48.128496853 (545h52m47.999579898s) [noise=true , address=77.90.31.13:4626 , piece_id=57EBKKWM5LSLSCDJTDYGCGLJHLEHUKAVQWOZSECM666SMRPJGLSA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:48.148997330 (545h52m48.020080375s) [noise=true , address=77.90.31.11:4554 , piece_id=AT2VTWBHTAGU27QADGLEXISPWJCP2YHJTZ3AY4QB7XTRCFMMBL3A, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:48.149021727 (545h52m48.020104762s) [seg_pos={0 0}, upload=478681]: Segment reached optimal threshold of 13 pieces. Dec 28 09:21:48.149271833 (545h52m48.020354878s) [noise=true , address=77.90.31.17:4770 , piece_id=NBCJK3Y4IN5RX6MX6T52MFQOQOGQV43CSHGGDXTOPNADU4UJKXHA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 09:21:48.149281240 (545h52m48.020364285s) [noise=true , address=77.90.31.17:4770 , piece_id=NBCJK3Y4IN5RX6MX6T52MFQOQOGQV43CSHGGDXTOPNADU4UJKXHA, seg_pos={0 0}, upload=478681]: Long tail cancellation due to optimal threshold Dec 28 09:21:48.149405868 (545h52m48.020488903s) [noise=true , address=77.90.31.19:4844 , piece_id=OH34RGEN7WKPC5IYY3K7ULZH6GGXET3BG4C6SSWO2OX24OSGXRDA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:58996->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:58996->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58996->77.90.31.19:4844: use of closed network connection Dec 28 09:21:48.149421738 (545h52m48.020504773s) [noise=true , address=77.90.31.19:4844 , piece_id=OH34RGEN7WKPC5IYY3K7ULZH6GGXET3BG4C6SSWO2OX24OSGXRDA, seg_pos={0 0}, upload=478681]: Long tail cancellation due to optimal threshold Dec 28 09:21:48.149709696 (545h52m48.020792731s) [noise=true , address=45.91.116.13:4153 , piece_id=KOACV2JXD72NTUFIRE6GXQRZLOPW4RWAOP5OFRK2FJKITMXMAKNQ, seg_pos={0 0}, upload=478681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 09:21:48.149717702 (545h52m48.020800747s) [noise=true , address=45.91.116.13:4153 , piece_id=KOACV2JXD72NTUFIRE6GXQRZLOPW4RWAOP5OFRK2FJKITMXMAKNQ, seg_pos={0 0}, upload=478681]: Long tail cancellation due to optimal threshold Dec 28 09:21:48.149755844 (545h52m48.02083888s) [seg_pos={0 0}, upload=478681]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:21:48.149915087 (545h52m48.020998132s) [upload=478681]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:21:48.149921259 (545h52m48.021004294s) [upload=478681]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:21:48.449306119 (545h52m48.320389164s) [upload=478681]: Done uploading segments... Dec 28 09:21:48.449329192 (545h52m48.320412237s) [upload=478681]: Upload finished. err: <nil> Duration: 6.026033769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X..X......XXXX.....XX.X.....X......X..............X....X] duration: 30.002350827s trace: https://jaeger.dev.storj.io/trace/1365710d528f2261 FAIL (2025-12-28 09:22:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 09:21:45.602958640 (545h52m45.474041685s) [upload=478681]: Starting upload Dec 28 09:21:45.603049033 (545h52m45.474132078s) [upload=478681]: Done starting upload Dec 28 09:21:45.603105449 (545h52m45.474188495s) [upload=478681]: Uploading segments... Dec 28 09:21:45.603310941 (545h52m45.474393986s) [upload=478681]: Got next segment. Inline: false Dec 28 09:21:45.603350326 (545h52m45.474433361s) [upload=478681]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:21:45.603355666 (545h52m45.474438711s) [upload=478681]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:21:45.851113360 (545h52m45.722196405s) [seg_pos={0 0}, upload=478681]: Begin upload segment... Dec 28 09:21:45.851184555 (545h52m45.72226759s) [seg_pos={0 0}, upload=478681]: Done begin upload segment. Dec 28 09:21:45.851241674 (545h52m45.722324709s) [noise=true , address=145.79.227.11:4730 , piece_id=JA6CXABHZ5WWMV4RDP6KOV4UWA6VUDRB5QHFTFH7B5SF6WAXYHTA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.851250341 (545h52m45.722333406s) [noise=true , address=145.79.227.7:4295 , piece_id=JGHDNGCXX34W2EKWJ7NJWFNB6PSCMZLO25CUFSZ3AVT34ACLL3SQ, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.851949492 (545h52m45.723032587s) [noise=true , address=145.79.227.34:4905 , piece_id=OO5M4G7PB3QCX5HNXFATZQZPPFJ5D77VPTPPQQAP3HO4ZFSCS75A, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.852631200 (545h52m45.723714245s) [noise=true , address=145.79.227.6:4258 , piece_id=UTZ73VCSMKBBW5XKHTZVBIGTYU5Z4OEW6NX3NFTMK5BNTCO2ZZKQ, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.853444759 (545h52m45.724527804s) [noise=true , address=145.79.227.5:4221 , piece_id=WTLYKNU3AOEAQ6HBFVI44BXQG4J4LDCSPAXNM3PYE75GZL7PZMJA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.854012841 (545h52m45.725095886s) [noise=true , address=77.90.31.17:4770 , piece_id=NBCJK3Y4IN5RX6MX6T52MFQOQOGQV43CSHGGDXTOPNADU4UJKXHA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.854506450 (545h52m45.725589496s) [noise=true , address=145.79.227.39:4015 , piece_id=TLIY2RKO2NCJBZY7FVBZTDGU6UUW4SUPQSU5ZOJ4TMW5WKT4D23Q, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.855528596 (545h52m45.726611631s) [noise=true , address=77.90.31.11:4554 , piece_id=AT2VTWBHTAGU27QADGLEXISPWJCP2YHJTZ3AY4QB7XTRCFMMBL3A, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.856150761 (545h52m45.727233806s) [noise=true , address=145.79.227.16:4910 , piece_id=DP6DFFELJNGLE3TWKRHZL5HKCFKWJ4BQ5M5IT7MXGOYF5NPKTC6Q, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.857032962 (545h52m45.728115997s) [noise=true , address=45.91.116.12:4227 , piece_id=HFRQERH3CDDHZXGDZAVKICZKWQP2OWWK7NVUMC2ECNYEMMF4U53Q, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.858329170 (545h52m45.729412215s) [noise=true , address=77.90.31.13:4626 , piece_id=57EBKKWM5LSLSCDJTDYGCGLJHLEHUKAVQWOZSECM666SMRPJGLSA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.858476059 (545h52m45.729559104s) [noise=true , address=145.79.227.2:4113 , piece_id=YKRAXDV5JRXHB3RMLCSE62W4V6KTGCCS5I3SAIOJXCMPD64MFNCA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.859961638 (545h52m45.731044673s) [noise=true , address=145.79.227.14:4838 , piece_id=7TVKHNJANF66BA3Q3LHOK6LRRNIKR7YYGKSIXNSASMIMFNGXH3PA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.860087317 (545h52m45.731170352s) [noise=true , address=45.91.116.13:4153 , piece_id=KOACV2JXD72NTUFIRE6GXQRZLOPW4RWAOP5OFRK2FJKITMXMAKNQ, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.861968259 (545h52m45.733051314s) [noise=true , address=77.90.31.19:4844 , piece_id=OH34RGEN7WKPC5IYY3K7ULZH6GGXET3BG4C6SSWO2OX24OSGXRDA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:45.862533406 (545h52m45.733616441s) [noise=true , address=45.91.116.8:4047 , piece_id=SNFCR7XN3ZNVF6YSCPAZKP3AB2T3QKSEFS2WMOJXP5HN3GV2LSJA, seg_pos={0 0}, upload=478681]: Uploading piece... Dec 28 09:21:47.579080171 (545h52m47.450163206s) [upload=478681]: Next returned nil segment Dec 28 09:21:47.579092003 (545h52m47.450175038s) [upload=478681]: Waiting for error group managing segments... Dec 28 09:21:47.858894054 (545h52m47.729977099s) [noise=true , address=145.79.227.11:4730 , piece_id=JA6CXABHZ5WWMV4RDP6KOV4UWA6VUDRB5QHFTFH7B5SF6WAXYHTA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.860354184 (545h52m47.731437229s) [noise=true , address=145.79.227.5:4221 , piece_id=WTLYKNU3AOEAQ6HBFVI44BXQG4J4LDCSPAXNM3PYE75GZL7PZMJA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.862538885 (545h52m47.73362192s) [noise=true , address=145.79.227.2:4113 , piece_id=YKRAXDV5JRXHB3RMLCSE62W4V6KTGCCS5I3SAIOJXCMPD64MFNCA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.871318562 (545h52m47.742401597s) [noise=true , address=145.79.227.14:4838 , piece_id=7TVKHNJANF66BA3Q3LHOK6LRRNIKR7YYGKSIXNSASMIMFNGXH3PA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.882626314 (545h52m47.75370936s) [noise=true , address=145.79.227.7:4295 , piece_id=JGHDNGCXX34W2EKWJ7NJWFNB6PSCMZLO25CUFSZ3AVT34ACLL3SQ, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.884550719 (545h52m47.755633764s) [noise=true , address=145.79.227.16:4910 , piece_id=DP6DFFELJNGLE3TWKRHZL5HKCFKWJ4BQ5M5IT7MXGOYF5NPKTC6Q, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.889669025 (545h52m47.76075206s) [noise=true , address=145.79.227.6:4258 , piece_id=UTZ73VCSMKBBW5XKHTZVBIGTYU5Z4OEW6NX3NFTMK5BNTCO2ZZKQ, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.892803403 (545h52m47.763886438s) [noise=true , address=145.79.227.39:4015 , piece_id=TLIY2RKO2NCJBZY7FVBZTDGU6UUW4SUPQSU5ZOJ4TMW5WKT4D23Q, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:47.967012180 (545h52m47.838095225s) [noise=true , address=145.79.227.34:4905 , piece_id=OO5M4G7PB3QCX5HNXFATZQZPPFJ5D77VPTPPQQAP3HO4ZFSCS75A, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:48.090280493 (545h52m47.961363528s) [noise=true , address=45.91.116.8:4047 , piece_id=SNFCR7XN3ZNVF6YSCPAZKP3AB2T3QKSEFS2WMOJXP5HN3GV2LSJA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:48.117585636 (545h52m47.988668691s) [noise=true , address=45.91.116.12:4227 , piece_id=HFRQERH3CDDHZXGDZAVKICZKWQP2OWWK7NVUMC2ECNYEMMF4U53Q, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:48.128496853 (545h52m47.999579898s) [noise=true , address=77.90.31.13:4626 , piece_id=57EBKKWM5LSLSCDJTDYGCGLJHLEHUKAVQWOZSECM666SMRPJGLSA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:48.148997330 (545h52m48.020080375s) [noise=true , address=77.90.31.11:4554 , piece_id=AT2VTWBHTAGU27QADGLEXISPWJCP2YHJTZ3AY4QB7XTRCFMMBL3A, seg_pos={0 0}, upload=478681]: Done uploading piece. err: <nil> Dec 28 09:21:48.149021727 (545h52m48.020104762s) [seg_pos={0 0}, upload=478681]: Segment reached optimal threshold of 13 pieces. Dec 28 09:21:48.149271833 (545h52m48.020354878s) [noise=true , address=77.90.31.17:4770 , piece_id=NBCJK3Y4IN5RX6MX6T52MFQOQOGQV43CSHGGDXTOPNADU4UJKXHA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 09:21:48.149281240 (545h52m48.020364285s) [noise=true , address=77.90.31.17:4770 , piece_id=NBCJK3Y4IN5RX6MX6T52MFQOQOGQV43CSHGGDXTOPNADU4UJKXHA, seg_pos={0 0}, upload=478681]: Long tail cancellation due to optimal threshold Dec 28 09:21:48.149405868 (545h52m48.020488903s) [noise=true , address=77.90.31.19:4844 , piece_id=OH34RGEN7WKPC5IYY3K7ULZH6GGXET3BG4C6SSWO2OX24OSGXRDA, seg_pos={0 0}, upload=478681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:58996->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:58996->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58996->77.90.31.19:4844: use of closed network connection Dec 28 09:21:48.149421738 (545h52m48.020504773s) [noise=true , address=77.90.31.19:4844 , piece_id=OH34RGEN7WKPC5IYY3K7ULZH6GGXET3BG4C6SSWO2OX24OSGXRDA, seg_pos={0 0}, upload=478681]: Long tail cancellation due to optimal threshold Dec 28 09:21:48.149709696 (545h52m48.020792731s) [noise=true , address=45.91.116.13:4153 , piece_id=KOACV2JXD72NTUFIRE6GXQRZLOPW4RWAOP5OFRK2FJKITMXMAKNQ, seg_pos={0 0}, upload=478681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 09:21:48.149717702 (545h52m48.020800747s) [noise=true , address=45.91.116.13:4153 , piece_id=KOACV2JXD72NTUFIRE6GXQRZLOPW4RWAOP5OFRK2FJKITMXMAKNQ, seg_pos={0 0}, upload=478681]: Long tail cancellation due to optimal threshold Dec 28 09:21:48.149755844 (545h52m48.02083888s) [seg_pos={0 0}, upload=478681]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:21:48.149915087 (545h52m48.020998132s) [upload=478681]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:21:48.149921259 (545h52m48.021004294s) [upload=478681]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:21:48.449306119 (545h52m48.320389164s) [upload=478681]: Done uploading segments... Dec 28 09:21:48.449329192 (545h52m48.320412237s) [upload=478681]: Upload finished. err: <nil> Duration: 30.0023941s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X....X.XX..XX..X........X.X...X....X.X.....X.....XXX.XX...XX] duration: 5.000920869s trace: https://jaeger.dev.storj.io/trace/3625f43800d4cd1e FAIL (2025-12-28 09:21:52 UTC) orbiter warning: context deadline exceeded Duration: 5.000931729s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.X....X.XX..XX..X........X.X...X....X.X.....X.....XXX.XX...X] duration: 5.001222745s trace: https://jaeger.dev.storj.io/trace/731d26f9087d4f00 FAIL (2025-12-28 09:21:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001239646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X..X......XXXX.....XX.X.....X......X..............X....] duration: 5.786112158s trace: https://jaeger.dev.storj.io/trace/18a527386a98a666 FAIL (2025-12-28 09:20:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q) context deadline exceeded ======LOGS====== ec 28 09:17:14.646142730 (545h48m14.517225765s) [upload=478617]: Starting upload Dec 28 09:17:14.646269822 (545h48m14.517352867s) [upload=478617]: Done starting upload Dec 28 09:17:14.646382607 (545h48m14.517465652s) [upload=478617]: Uploading segments... Dec 28 09:17:14.646461056 (545h48m14.517544091s) [upload=478617]: Got next segment. Inline: false Dec 28 09:17:14.647059887 (545h48m14.518142932s) [upload=478617]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:17:14.647066639 (545h48m14.518149675s) [upload=478617]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:17:14.895463028 (545h48m14.766546063s) [seg_pos={0 0}, upload=478617]: Begin upload segment... Dec 28 09:17:14.895510769 (545h48m14.766593804s) [seg_pos={0 0}, upload=478617]: Done begin upload segment. Dec 28 09:17:14.895536298 (545h48m14.766619343s) [noise=true , address=145.79.227.47:4266 , piece_id=3B32OOXLO2XFPDFPUPWD63MFRQXOFAQMXNPKX6CC7KASVXJRE54A, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896230269 (545h48m14.767313314s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896390855 (545h48m14.7674739s) [noise=true , address=145.79.227.45:4230 , piece_id=OMTIPMJPIGIYMVF3WRHITWOWFVYGGJ4MZUXTT4CJVIZ2P36Z34UA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896746672 (545h48m14.767829707s) [noise=true , address=145.79.227.14:4838 , piece_id=X2RHCXHDIFMA2ENAOMCBWH6VQIKVMOMZXIUTOQ3ABDEKSHDJMTYA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897007018 (545h48m14.768090053s) [noise=true , address=45.91.116.5:4297 , piece_id=SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897401168 (545h48m14.768484223s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897456364 (545h48m14.768539399s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897767757 (545h48m14.768850792s) [noise=true , address=45.91.116.10:4081 , piece_id=TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898089229 (545h48m14.769172264s) [noise=true , address=145.79.227.48:4301 , piece_id=KNXU2A77GZ7SPGH6THLVCISROEFUI3NZ7XAZMRINBGMLOFIXXPHQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898102704 (545h48m14.769185739s) [noise=true , address=45.91.116.4:4940 , piece_id=EUK5DFYAMPOTZKKQ27QMJLFBYEC74IJ4ETU4XR2OKUHQPBWYRNOA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898649996 (545h48m14.769733031s) [noise=true , address=145.79.227.51:4409 , piece_id=ARCTECRUUY2U5FSUDM7OXC2CEXZJO6SNTEX2SNIFTMM2RXELUFLQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898811945 (545h48m14.76989499s) [noise=true , address=145.79.227.37:4567 , piece_id=6ROLV67SCGVXL2NROK74CJITQEXIAQLEAH2XKLB7SXUTOBTOR7FA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.899402760 (545h48m14.770485795s) [noise=true , address=145.79.227.18:4007 , piece_id=5IDHZHUS5ZEM7FBVMBEOBEXOWOHZGONRYAZ463USDUNK4KG3R6SQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.900028121 (545h48m14.771111156s) [noise=true , address=145.79.227.16:4911 , piece_id=DR6ZIDW3RZWHQ2LKJMRWGGFBYYNJ5J5EABFSWXMGWIDK3RCDQ42Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.900588187 (545h48m14.771671232s) [noise=true , address=145.79.227.27:4689 , piece_id=55Q7U6KFSNU67CX722SY5PNMDTH4ER7APDGH7BKAJGUP6NP3PQGA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.901054665 (545h48m14.7721377s) [noise=true , address=145.79.227.29:4942 , piece_id=BEI5RIBBQZVGE3XSLPX266GUIQBE265DRP7NLV7ORZIJVIGDM54Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:16.614783937 (545h48m16.485866982s) [upload=478617]: Next returned nil segment Dec 28 09:17:16.614799506 (545h48m16.485882541s) [upload=478617]: Waiting for error group managing segments... Dec 28 09:17:16.903181063 (545h48m16.774264098s) [noise=true , address=145.79.227.18:4007 , piece_id=5IDHZHUS5ZEM7FBVMBEOBEXOWOHZGONRYAZ463USDUNK4KG3R6SQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.906044847 (545h48m16.777127892s) [noise=true , address=145.79.227.37:4567 , piece_id=6ROLV67SCGVXL2NROK74CJITQEXIAQLEAH2XKLB7SXUTOBTOR7FA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.910705191 (545h48m16.781788236s) [noise=true , address=145.79.227.14:4838 , piece_id=X2RHCXHDIFMA2ENAOMCBWH6VQIKVMOMZXIUTOQ3ABDEKSHDJMTYA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.917271434 (545h48m16.788354469s) [noise=true , address=145.79.227.45:4230 , piece_id=OMTIPMJPIGIYMVF3WRHITWOWFVYGGJ4MZUXTT4CJVIZ2P36Z34UA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.919809978 (545h48m16.790893013s) [noise=true , address=145.79.227.16:4911 , piece_id=DR6ZIDW3RZWHQ2LKJMRWGGFBYYNJ5J5EABFSWXMGWIDK3RCDQ42Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.920161106 (545h48m16.791244151s) [noise=true , address=145.79.227.48:4301 , piece_id=KNXU2A77GZ7SPGH6THLVCISROEFUI3NZ7XAZMRINBGMLOFIXXPHQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.923630453 (545h48m16.794713488s) [noise=true , address=145.79.227.27:4689 , piece_id=55Q7U6KFSNU67CX722SY5PNMDTH4ER7APDGH7BKAJGUP6NP3PQGA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.926877115 (545h48m16.79796016s) [noise=true , address=145.79.227.51:4409 , piece_id=ARCTECRUUY2U5FSUDM7OXC2CEXZJO6SNTEX2SNIFTMM2RXELUFLQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.927652300 (545h48m16.798735345s) [noise=true , address=145.79.227.29:4942 , piece_id=BEI5RIBBQZVGE3XSLPX266GUIQBE265DRP7NLV7ORZIJVIGDM54Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.932476477 (545h48m16.803559512s) [noise=true , address=145.79.227.47:4266 , piece_id=3B32OOXLO2XFPDFPUPWD63MFRQXOFAQMXNPKX6CC7KASVXJRE54A, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.137987024 (545h48m17.009070059s) [noise=true , address=45.91.116.5:4297 , piece_id=SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.162012623 (545h48m17.033095658s) [noise=true , address=45.91.116.10:4081 , piece_id=TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.170451531 (545h48m17.041534566s) [noise=true , address=45.91.116.4:4940 , piece_id=EUK5DFYAMPOTZKKQ27QMJLFBYEC74IJ4ETU4XR2OKUHQPBWYRNOA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.170477551 (545h48m17.041560596s) [seg_pos={0 0}, upload=478617]: Segment reached optimal threshold of 13 pieces. Dec 28 09:17:17.170658676 (545h48m17.041741711s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 09:17:17.170665038 (545h48m17.041748083s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.170887782 (545h48m17.041970817s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 09:17:17.170893714 (545h48m17.041976759s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.171022088 (545h48m17.042105133s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection Dec 28 09:17:17.171028090 (545h48m17.042111125s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.171044641 (545h48m17.042127676s) [seg_pos={0 0}, upload=478617]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:17:17.171109604 (545h48m17.042192649s) [upload=478617]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:17:17.171113993 (545h48m17.042197038s) [upload=478617]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:17:17.460388621 (545h48m17.331471656s) [upload=478617]: Done uploading segments... Dec 28 09:17:17.460403749 (545h48m17.331486784s) [upload=478617]: Upload finished. err: <nil> Duration: 5.786140561s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..X....X.XX..XX..X........X.X...X....X.X.....X.....XXX.XX...] duration: 5.000769663s trace: https://jaeger.dev.storj.io/trace/bbab6bbfce78909 FAIL (2025-12-28 09:20:34 UTC) orbiter warning: context deadline exceeded Duration: 5.000800491s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...X....X.XX..XX..X........X.X...X....X.X.....X.....XXX.XX..] duration: 5.001369155s trace: https://jaeger.dev.storj.io/trace/bbab6bbfce78909 FAIL (2025-12-28 09:19:59 UTC) orbiter warning: context deadline exceeded Duration: 5.001420753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X..X......XXXX.....XX.X.....X......X..............X...] duration: 5.903568043s trace: https://jaeger.dev.storj.io/trace/18a527386a98a666 FAIL (2025-12-28 09:19:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q) context deadline exceeded ======LOGS====== ec 28 09:17:14.646142730 (545h48m14.517225765s) [upload=478617]: Starting upload Dec 28 09:17:14.646269822 (545h48m14.517352867s) [upload=478617]: Done starting upload Dec 28 09:17:14.646382607 (545h48m14.517465652s) [upload=478617]: Uploading segments... Dec 28 09:17:14.646461056 (545h48m14.517544091s) [upload=478617]: Got next segment. Inline: false Dec 28 09:17:14.647059887 (545h48m14.518142932s) [upload=478617]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:17:14.647066639 (545h48m14.518149675s) [upload=478617]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:17:14.895463028 (545h48m14.766546063s) [seg_pos={0 0}, upload=478617]: Begin upload segment... Dec 28 09:17:14.895510769 (545h48m14.766593804s) [seg_pos={0 0}, upload=478617]: Done begin upload segment. Dec 28 09:17:14.895536298 (545h48m14.766619343s) [noise=true , address=145.79.227.47:4266 , piece_id=3B32OOXLO2XFPDFPUPWD63MFRQXOFAQMXNPKX6CC7KASVXJRE54A, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896230269 (545h48m14.767313314s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896390855 (545h48m14.7674739s) [noise=true , address=145.79.227.45:4230 , piece_id=OMTIPMJPIGIYMVF3WRHITWOWFVYGGJ4MZUXTT4CJVIZ2P36Z34UA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896746672 (545h48m14.767829707s) [noise=true , address=145.79.227.14:4838 , piece_id=X2RHCXHDIFMA2ENAOMCBWH6VQIKVMOMZXIUTOQ3ABDEKSHDJMTYA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897007018 (545h48m14.768090053s) [noise=true , address=45.91.116.5:4297 , piece_id=SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897401168 (545h48m14.768484223s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897456364 (545h48m14.768539399s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897767757 (545h48m14.768850792s) [noise=true , address=45.91.116.10:4081 , piece_id=TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898089229 (545h48m14.769172264s) [noise=true , address=145.79.227.48:4301 , piece_id=KNXU2A77GZ7SPGH6THLVCISROEFUI3NZ7XAZMRINBGMLOFIXXPHQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898102704 (545h48m14.769185739s) [noise=true , address=45.91.116.4:4940 , piece_id=EUK5DFYAMPOTZKKQ27QMJLFBYEC74IJ4ETU4XR2OKUHQPBWYRNOA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898649996 (545h48m14.769733031s) [noise=true , address=145.79.227.51:4409 , piece_id=ARCTECRUUY2U5FSUDM7OXC2CEXZJO6SNTEX2SNIFTMM2RXELUFLQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898811945 (545h48m14.76989499s) [noise=true , address=145.79.227.37:4567 , piece_id=6ROLV67SCGVXL2NROK74CJITQEXIAQLEAH2XKLB7SXUTOBTOR7FA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.899402760 (545h48m14.770485795s) [noise=true , address=145.79.227.18:4007 , piece_id=5IDHZHUS5ZEM7FBVMBEOBEXOWOHZGONRYAZ463USDUNK4KG3R6SQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.900028121 (545h48m14.771111156s) [noise=true , address=145.79.227.16:4911 , piece_id=DR6ZIDW3RZWHQ2LKJMRWGGFBYYNJ5J5EABFSWXMGWIDK3RCDQ42Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.900588187 (545h48m14.771671232s) [noise=true , address=145.79.227.27:4689 , piece_id=55Q7U6KFSNU67CX722SY5PNMDTH4ER7APDGH7BKAJGUP6NP3PQGA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.901054665 (545h48m14.7721377s) [noise=true , address=145.79.227.29:4942 , piece_id=BEI5RIBBQZVGE3XSLPX266GUIQBE265DRP7NLV7ORZIJVIGDM54Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:16.614783937 (545h48m16.485866982s) [upload=478617]: Next returned nil segment Dec 28 09:17:16.614799506 (545h48m16.485882541s) [upload=478617]: Waiting for error group managing segments... Dec 28 09:17:16.903181063 (545h48m16.774264098s) [noise=true , address=145.79.227.18:4007 , piece_id=5IDHZHUS5ZEM7FBVMBEOBEXOWOHZGONRYAZ463USDUNK4KG3R6SQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.906044847 (545h48m16.777127892s) [noise=true , address=145.79.227.37:4567 , piece_id=6ROLV67SCGVXL2NROK74CJITQEXIAQLEAH2XKLB7SXUTOBTOR7FA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.910705191 (545h48m16.781788236s) [noise=true , address=145.79.227.14:4838 , piece_id=X2RHCXHDIFMA2ENAOMCBWH6VQIKVMOMZXIUTOQ3ABDEKSHDJMTYA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.917271434 (545h48m16.788354469s) [noise=true , address=145.79.227.45:4230 , piece_id=OMTIPMJPIGIYMVF3WRHITWOWFVYGGJ4MZUXTT4CJVIZ2P36Z34UA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.919809978 (545h48m16.790893013s) [noise=true , address=145.79.227.16:4911 , piece_id=DR6ZIDW3RZWHQ2LKJMRWGGFBYYNJ5J5EABFSWXMGWIDK3RCDQ42Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.920161106 (545h48m16.791244151s) [noise=true , address=145.79.227.48:4301 , piece_id=KNXU2A77GZ7SPGH6THLVCISROEFUI3NZ7XAZMRINBGMLOFIXXPHQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.923630453 (545h48m16.794713488s) [noise=true , address=145.79.227.27:4689 , piece_id=55Q7U6KFSNU67CX722SY5PNMDTH4ER7APDGH7BKAJGUP6NP3PQGA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.926877115 (545h48m16.79796016s) [noise=true , address=145.79.227.51:4409 , piece_id=ARCTECRUUY2U5FSUDM7OXC2CEXZJO6SNTEX2SNIFTMM2RXELUFLQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.927652300 (545h48m16.798735345s) [noise=true , address=145.79.227.29:4942 , piece_id=BEI5RIBBQZVGE3XSLPX266GUIQBE265DRP7NLV7ORZIJVIGDM54Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.932476477 (545h48m16.803559512s) [noise=true , address=145.79.227.47:4266 , piece_id=3B32OOXLO2XFPDFPUPWD63MFRQXOFAQMXNPKX6CC7KASVXJRE54A, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.137987024 (545h48m17.009070059s) [noise=true , address=45.91.116.5:4297 , piece_id=SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.162012623 (545h48m17.033095658s) [noise=true , address=45.91.116.10:4081 , piece_id=TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.170451531 (545h48m17.041534566s) [noise=true , address=45.91.116.4:4940 , piece_id=EUK5DFYAMPOTZKKQ27QMJLFBYEC74IJ4ETU4XR2OKUHQPBWYRNOA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.170477551 (545h48m17.041560596s) [seg_pos={0 0}, upload=478617]: Segment reached optimal threshold of 13 pieces. Dec 28 09:17:17.170658676 (545h48m17.041741711s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 09:17:17.170665038 (545h48m17.041748083s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.170887782 (545h48m17.041970817s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 09:17:17.170893714 (545h48m17.041976759s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.171022088 (545h48m17.042105133s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection Dec 28 09:17:17.171028090 (545h48m17.042111125s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.171044641 (545h48m17.042127676s) [seg_pos={0 0}, upload=478617]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:17:17.171109604 (545h48m17.042192649s) [upload=478617]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:17:17.171113993 (545h48m17.042197038s) [upload=478617]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:17:17.460388621 (545h48m17.331471656s) [upload=478617]: Done uploading segments... Dec 28 09:17:17.460403749 (545h48m17.331486784s) [upload=478617]: Upload finished. err: <nil> Duration: 5.903594233s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [....X....X.XX..XX..X........X.X...X....X.X.....X.....XXX.XX.] duration: 3.556797535s trace: https://jaeger.dev.storj.io/trace/bbab6bbfce78909 FAIL (2025-12-28 09:19:27 UTC) orbiter warning: context deadline exceeded Duration: 3.556823295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X..X......XXXX.....XX.X.....X......X..............X..] duration: 5.642603232s trace: https://jaeger.dev.storj.io/trace/18a527386a98a666 FAIL (2025-12-28 09:18:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q) context deadline exceeded ======LOGS====== ec 28 09:17:14.646142730 (545h48m14.517225765s) [upload=478617]: Starting upload Dec 28 09:17:14.646269822 (545h48m14.517352867s) [upload=478617]: Done starting upload Dec 28 09:17:14.646382607 (545h48m14.517465652s) [upload=478617]: Uploading segments... Dec 28 09:17:14.646461056 (545h48m14.517544091s) [upload=478617]: Got next segment. Inline: false Dec 28 09:17:14.647059887 (545h48m14.518142932s) [upload=478617]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:17:14.647066639 (545h48m14.518149675s) [upload=478617]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:17:14.895463028 (545h48m14.766546063s) [seg_pos={0 0}, upload=478617]: Begin upload segment... Dec 28 09:17:14.895510769 (545h48m14.766593804s) [seg_pos={0 0}, upload=478617]: Done begin upload segment. Dec 28 09:17:14.895536298 (545h48m14.766619343s) [noise=true , address=145.79.227.47:4266 , piece_id=3B32OOXLO2XFPDFPUPWD63MFRQXOFAQMXNPKX6CC7KASVXJRE54A, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896230269 (545h48m14.767313314s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896390855 (545h48m14.7674739s) [noise=true , address=145.79.227.45:4230 , piece_id=OMTIPMJPIGIYMVF3WRHITWOWFVYGGJ4MZUXTT4CJVIZ2P36Z34UA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896746672 (545h48m14.767829707s) [noise=true , address=145.79.227.14:4838 , piece_id=X2RHCXHDIFMA2ENAOMCBWH6VQIKVMOMZXIUTOQ3ABDEKSHDJMTYA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897007018 (545h48m14.768090053s) [noise=true , address=45.91.116.5:4297 , piece_id=SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897401168 (545h48m14.768484223s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897456364 (545h48m14.768539399s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897767757 (545h48m14.768850792s) [noise=true , address=45.91.116.10:4081 , piece_id=TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898089229 (545h48m14.769172264s) [noise=true , address=145.79.227.48:4301 , piece_id=KNXU2A77GZ7SPGH6THLVCISROEFUI3NZ7XAZMRINBGMLOFIXXPHQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898102704 (545h48m14.769185739s) [noise=true , address=45.91.116.4:4940 , piece_id=EUK5DFYAMPOTZKKQ27QMJLFBYEC74IJ4ETU4XR2OKUHQPBWYRNOA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898649996 (545h48m14.769733031s) [noise=true , address=145.79.227.51:4409 , piece_id=ARCTECRUUY2U5FSUDM7OXC2CEXZJO6SNTEX2SNIFTMM2RXELUFLQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898811945 (545h48m14.76989499s) [noise=true , address=145.79.227.37:4567 , piece_id=6ROLV67SCGVXL2NROK74CJITQEXIAQLEAH2XKLB7SXUTOBTOR7FA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.899402760 (545h48m14.770485795s) [noise=true , address=145.79.227.18:4007 , piece_id=5IDHZHUS5ZEM7FBVMBEOBEXOWOHZGONRYAZ463USDUNK4KG3R6SQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.900028121 (545h48m14.771111156s) [noise=true , address=145.79.227.16:4911 , piece_id=DR6ZIDW3RZWHQ2LKJMRWGGFBYYNJ5J5EABFSWXMGWIDK3RCDQ42Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.900588187 (545h48m14.771671232s) [noise=true , address=145.79.227.27:4689 , piece_id=55Q7U6KFSNU67CX722SY5PNMDTH4ER7APDGH7BKAJGUP6NP3PQGA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.901054665 (545h48m14.7721377s) [noise=true , address=145.79.227.29:4942 , piece_id=BEI5RIBBQZVGE3XSLPX266GUIQBE265DRP7NLV7ORZIJVIGDM54Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:16.614783937 (545h48m16.485866982s) [upload=478617]: Next returned nil segment Dec 28 09:17:16.614799506 (545h48m16.485882541s) [upload=478617]: Waiting for error group managing segments... Dec 28 09:17:16.903181063 (545h48m16.774264098s) [noise=true , address=145.79.227.18:4007 , piece_id=5IDHZHUS5ZEM7FBVMBEOBEXOWOHZGONRYAZ463USDUNK4KG3R6SQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.906044847 (545h48m16.777127892s) [noise=true , address=145.79.227.37:4567 , piece_id=6ROLV67SCGVXL2NROK74CJITQEXIAQLEAH2XKLB7SXUTOBTOR7FA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.910705191 (545h48m16.781788236s) [noise=true , address=145.79.227.14:4838 , piece_id=X2RHCXHDIFMA2ENAOMCBWH6VQIKVMOMZXIUTOQ3ABDEKSHDJMTYA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.917271434 (545h48m16.788354469s) [noise=true , address=145.79.227.45:4230 , piece_id=OMTIPMJPIGIYMVF3WRHITWOWFVYGGJ4MZUXTT4CJVIZ2P36Z34UA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.919809978 (545h48m16.790893013s) [noise=true , address=145.79.227.16:4911 , piece_id=DR6ZIDW3RZWHQ2LKJMRWGGFBYYNJ5J5EABFSWXMGWIDK3RCDQ42Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.920161106 (545h48m16.791244151s) [noise=true , address=145.79.227.48:4301 , piece_id=KNXU2A77GZ7SPGH6THLVCISROEFUI3NZ7XAZMRINBGMLOFIXXPHQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.923630453 (545h48m16.794713488s) [noise=true , address=145.79.227.27:4689 , piece_id=55Q7U6KFSNU67CX722SY5PNMDTH4ER7APDGH7BKAJGUP6NP3PQGA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.926877115 (545h48m16.79796016s) [noise=true , address=145.79.227.51:4409 , piece_id=ARCTECRUUY2U5FSUDM7OXC2CEXZJO6SNTEX2SNIFTMM2RXELUFLQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.927652300 (545h48m16.798735345s) [noise=true , address=145.79.227.29:4942 , piece_id=BEI5RIBBQZVGE3XSLPX266GUIQBE265DRP7NLV7ORZIJVIGDM54Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.932476477 (545h48m16.803559512s) [noise=true , address=145.79.227.47:4266 , piece_id=3B32OOXLO2XFPDFPUPWD63MFRQXOFAQMXNPKX6CC7KASVXJRE54A, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.137987024 (545h48m17.009070059s) [noise=true , address=45.91.116.5:4297 , piece_id=SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.162012623 (545h48m17.033095658s) [noise=true , address=45.91.116.10:4081 , piece_id=TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.170451531 (545h48m17.041534566s) [noise=true , address=45.91.116.4:4940 , piece_id=EUK5DFYAMPOTZKKQ27QMJLFBYEC74IJ4ETU4XR2OKUHQPBWYRNOA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.170477551 (545h48m17.041560596s) [seg_pos={0 0}, upload=478617]: Segment reached optimal threshold of 13 pieces. Dec 28 09:17:17.170658676 (545h48m17.041741711s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 09:17:17.170665038 (545h48m17.041748083s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.170887782 (545h48m17.041970817s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 09:17:17.170893714 (545h48m17.041976759s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.171022088 (545h48m17.042105133s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection Dec 28 09:17:17.171028090 (545h48m17.042111125s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.171044641 (545h48m17.042127676s) [seg_pos={0 0}, upload=478617]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:17:17.171109604 (545h48m17.042192649s) [upload=478617]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:17:17.171113993 (545h48m17.042197038s) [upload=478617]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:17:17.460388621 (545h48m17.331471656s) [upload=478617]: Done uploading segments... Dec 28 09:17:17.460403749 (545h48m17.331486784s) [upload=478617]: Upload finished. err: <nil> Duration: 5.64263384s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [X....X....X.XX..XX..X........X.X...X....X.X.....X.....XXX.XX] duration: 5.001810636s trace: https://jaeger.dev.storj.io/trace/bbab6bbfce78909 FAIL (2025-12-28 09:18:21 UTC) orbiter warning: context deadline exceeded Duration: 5.001837326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.......X..X......XXXX.....XX.X.....X......X..............X.] duration: 6.012852669s trace: https://jaeger.dev.storj.io/trace/18a527386a98a666 FAIL (2025-12-28 09:18:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q) context deadline exceeded ======LOGS====== ec 28 09:17:14.646142730 (545h48m14.517225765s) [upload=478617]: Starting upload Dec 28 09:17:14.646269822 (545h48m14.517352867s) [upload=478617]: Done starting upload Dec 28 09:17:14.646382607 (545h48m14.517465652s) [upload=478617]: Uploading segments... Dec 28 09:17:14.646461056 (545h48m14.517544091s) [upload=478617]: Got next segment. Inline: false Dec 28 09:17:14.647059887 (545h48m14.518142932s) [upload=478617]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:17:14.647066639 (545h48m14.518149675s) [upload=478617]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:17:14.895463028 (545h48m14.766546063s) [seg_pos={0 0}, upload=478617]: Begin upload segment... Dec 28 09:17:14.895510769 (545h48m14.766593804s) [seg_pos={0 0}, upload=478617]: Done begin upload segment. Dec 28 09:17:14.895536298 (545h48m14.766619343s) [noise=true , address=145.79.227.47:4266 , piece_id=3B32OOXLO2XFPDFPUPWD63MFRQXOFAQMXNPKX6CC7KASVXJRE54A, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896230269 (545h48m14.767313314s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896390855 (545h48m14.7674739s) [noise=true , address=145.79.227.45:4230 , piece_id=OMTIPMJPIGIYMVF3WRHITWOWFVYGGJ4MZUXTT4CJVIZ2P36Z34UA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896746672 (545h48m14.767829707s) [noise=true , address=145.79.227.14:4838 , piece_id=X2RHCXHDIFMA2ENAOMCBWH6VQIKVMOMZXIUTOQ3ABDEKSHDJMTYA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897007018 (545h48m14.768090053s) [noise=true , address=45.91.116.5:4297 , piece_id=SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897401168 (545h48m14.768484223s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897456364 (545h48m14.768539399s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897767757 (545h48m14.768850792s) [noise=true , address=45.91.116.10:4081 , piece_id=TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898089229 (545h48m14.769172264s) [noise=true , address=145.79.227.48:4301 , piece_id=KNXU2A77GZ7SPGH6THLVCISROEFUI3NZ7XAZMRINBGMLOFIXXPHQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898102704 (545h48m14.769185739s) [noise=true , address=45.91.116.4:4940 , piece_id=EUK5DFYAMPOTZKKQ27QMJLFBYEC74IJ4ETU4XR2OKUHQPBWYRNOA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898649996 (545h48m14.769733031s) [noise=true , address=145.79.227.51:4409 , piece_id=ARCTECRUUY2U5FSUDM7OXC2CEXZJO6SNTEX2SNIFTMM2RXELUFLQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898811945 (545h48m14.76989499s) [noise=true , address=145.79.227.37:4567 , piece_id=6ROLV67SCGVXL2NROK74CJITQEXIAQLEAH2XKLB7SXUTOBTOR7FA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.899402760 (545h48m14.770485795s) [noise=true , address=145.79.227.18:4007 , piece_id=5IDHZHUS5ZEM7FBVMBEOBEXOWOHZGONRYAZ463USDUNK4KG3R6SQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.900028121 (545h48m14.771111156s) [noise=true , address=145.79.227.16:4911 , piece_id=DR6ZIDW3RZWHQ2LKJMRWGGFBYYNJ5J5EABFSWXMGWIDK3RCDQ42Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.900588187 (545h48m14.771671232s) [noise=true , address=145.79.227.27:4689 , piece_id=55Q7U6KFSNU67CX722SY5PNMDTH4ER7APDGH7BKAJGUP6NP3PQGA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.901054665 (545h48m14.7721377s) [noise=true , address=145.79.227.29:4942 , piece_id=BEI5RIBBQZVGE3XSLPX266GUIQBE265DRP7NLV7ORZIJVIGDM54Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:16.614783937 (545h48m16.485866982s) [upload=478617]: Next returned nil segment Dec 28 09:17:16.614799506 (545h48m16.485882541s) [upload=478617]: Waiting for error group managing segments... Dec 28 09:17:16.903181063 (545h48m16.774264098s) [noise=true , address=145.79.227.18:4007 , piece_id=5IDHZHUS5ZEM7FBVMBEOBEXOWOHZGONRYAZ463USDUNK4KG3R6SQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.906044847 (545h48m16.777127892s) [noise=true , address=145.79.227.37:4567 , piece_id=6ROLV67SCGVXL2NROK74CJITQEXIAQLEAH2XKLB7SXUTOBTOR7FA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.910705191 (545h48m16.781788236s) [noise=true , address=145.79.227.14:4838 , piece_id=X2RHCXHDIFMA2ENAOMCBWH6VQIKVMOMZXIUTOQ3ABDEKSHDJMTYA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.917271434 (545h48m16.788354469s) [noise=true , address=145.79.227.45:4230 , piece_id=OMTIPMJPIGIYMVF3WRHITWOWFVYGGJ4MZUXTT4CJVIZ2P36Z34UA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.919809978 (545h48m16.790893013s) [noise=true , address=145.79.227.16:4911 , piece_id=DR6ZIDW3RZWHQ2LKJMRWGGFBYYNJ5J5EABFSWXMGWIDK3RCDQ42Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.920161106 (545h48m16.791244151s) [noise=true , address=145.79.227.48:4301 , piece_id=KNXU2A77GZ7SPGH6THLVCISROEFUI3NZ7XAZMRINBGMLOFIXXPHQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.923630453 (545h48m16.794713488s) [noise=true , address=145.79.227.27:4689 , piece_id=55Q7U6KFSNU67CX722SY5PNMDTH4ER7APDGH7BKAJGUP6NP3PQGA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.926877115 (545h48m16.79796016s) [noise=true , address=145.79.227.51:4409 , piece_id=ARCTECRUUY2U5FSUDM7OXC2CEXZJO6SNTEX2SNIFTMM2RXELUFLQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.927652300 (545h48m16.798735345s) [noise=true , address=145.79.227.29:4942 , piece_id=BEI5RIBBQZVGE3XSLPX266GUIQBE265DRP7NLV7ORZIJVIGDM54Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.932476477 (545h48m16.803559512s) [noise=true , address=145.79.227.47:4266 , piece_id=3B32OOXLO2XFPDFPUPWD63MFRQXOFAQMXNPKX6CC7KASVXJRE54A, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.137987024 (545h48m17.009070059s) [noise=true , address=45.91.116.5:4297 , piece_id=SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.162012623 (545h48m17.033095658s) [noise=true , address=45.91.116.10:4081 , piece_id=TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.170451531 (545h48m17.041534566s) [noise=true , address=45.91.116.4:4940 , piece_id=EUK5DFYAMPOTZKKQ27QMJLFBYEC74IJ4ETU4XR2OKUHQPBWYRNOA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.170477551 (545h48m17.041560596s) [seg_pos={0 0}, upload=478617]: Segment reached optimal threshold of 13 pieces. Dec 28 09:17:17.170658676 (545h48m17.041741711s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 09:17:17.170665038 (545h48m17.041748083s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.170887782 (545h48m17.041970817s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 09:17:17.170893714 (545h48m17.041976759s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.171022088 (545h48m17.042105133s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection Dec 28 09:17:17.171028090 (545h48m17.042111125s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.171044641 (545h48m17.042127676s) [seg_pos={0 0}, upload=478617]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:17:17.171109604 (545h48m17.042192649s) [upload=478617]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:17:17.171113993 (545h48m17.042197038s) [upload=478617]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:17:17.460388621 (545h48m17.331471656s) [upload=478617]: Done uploading segments... Dec 28 09:17:17.460403749 (545h48m17.331486784s) [upload=478617]: Upload finished. err: <nil> Duration: 6.012881845s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XX....X....X.XX..XX..X........X.X...X....X.X.....X.....XXX.X] duration: 5.000973082s trace: https://jaeger.dev.storj.io/trace/33473c16d83fefed FAIL (2025-12-28 09:17:45 UTC) orbiter warning: context deadline exceeded Duration: 5.000995193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX.......X..X......XXXX.....XX.X.....X......X..............X] duration: 30.003112831s trace: https://jaeger.dev.storj.io/trace/18a527386a98a666 FAIL (2025-12-28 09:17:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q) context deadline exceeded ======LOGS====== ec 28 09:17:14.646142730 (545h48m14.517225765s) [upload=478617]: Starting upload Dec 28 09:17:14.646269822 (545h48m14.517352867s) [upload=478617]: Done starting upload Dec 28 09:17:14.646382607 (545h48m14.517465652s) [upload=478617]: Uploading segments... Dec 28 09:17:14.646461056 (545h48m14.517544091s) [upload=478617]: Got next segment. Inline: false Dec 28 09:17:14.647059887 (545h48m14.518142932s) [upload=478617]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:17:14.647066639 (545h48m14.518149675s) [upload=478617]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:17:14.895463028 (545h48m14.766546063s) [seg_pos={0 0}, upload=478617]: Begin upload segment... Dec 28 09:17:14.895510769 (545h48m14.766593804s) [seg_pos={0 0}, upload=478617]: Done begin upload segment. Dec 28 09:17:14.895536298 (545h48m14.766619343s) [noise=true , address=145.79.227.47:4266 , piece_id=3B32OOXLO2XFPDFPUPWD63MFRQXOFAQMXNPKX6CC7KASVXJRE54A, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896230269 (545h48m14.767313314s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896390855 (545h48m14.7674739s) [noise=true , address=145.79.227.45:4230 , piece_id=OMTIPMJPIGIYMVF3WRHITWOWFVYGGJ4MZUXTT4CJVIZ2P36Z34UA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.896746672 (545h48m14.767829707s) [noise=true , address=145.79.227.14:4838 , piece_id=X2RHCXHDIFMA2ENAOMCBWH6VQIKVMOMZXIUTOQ3ABDEKSHDJMTYA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897007018 (545h48m14.768090053s) [noise=true , address=45.91.116.5:4297 , piece_id=SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897401168 (545h48m14.768484223s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897456364 (545h48m14.768539399s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.897767757 (545h48m14.768850792s) [noise=true , address=45.91.116.10:4081 , piece_id=TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898089229 (545h48m14.769172264s) [noise=true , address=145.79.227.48:4301 , piece_id=KNXU2A77GZ7SPGH6THLVCISROEFUI3NZ7XAZMRINBGMLOFIXXPHQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898102704 (545h48m14.769185739s) [noise=true , address=45.91.116.4:4940 , piece_id=EUK5DFYAMPOTZKKQ27QMJLFBYEC74IJ4ETU4XR2OKUHQPBWYRNOA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898649996 (545h48m14.769733031s) [noise=true , address=145.79.227.51:4409 , piece_id=ARCTECRUUY2U5FSUDM7OXC2CEXZJO6SNTEX2SNIFTMM2RXELUFLQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.898811945 (545h48m14.76989499s) [noise=true , address=145.79.227.37:4567 , piece_id=6ROLV67SCGVXL2NROK74CJITQEXIAQLEAH2XKLB7SXUTOBTOR7FA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.899402760 (545h48m14.770485795s) [noise=true , address=145.79.227.18:4007 , piece_id=5IDHZHUS5ZEM7FBVMBEOBEXOWOHZGONRYAZ463USDUNK4KG3R6SQ, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.900028121 (545h48m14.771111156s) [noise=true , address=145.79.227.16:4911 , piece_id=DR6ZIDW3RZWHQ2LKJMRWGGFBYYNJ5J5EABFSWXMGWIDK3RCDQ42Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.900588187 (545h48m14.771671232s) [noise=true , address=145.79.227.27:4689 , piece_id=55Q7U6KFSNU67CX722SY5PNMDTH4ER7APDGH7BKAJGUP6NP3PQGA, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:14.901054665 (545h48m14.7721377s) [noise=true , address=145.79.227.29:4942 , piece_id=BEI5RIBBQZVGE3XSLPX266GUIQBE265DRP7NLV7ORZIJVIGDM54Q, seg_pos={0 0}, upload=478617]: Uploading piece... Dec 28 09:17:16.614783937 (545h48m16.485866982s) [upload=478617]: Next returned nil segment Dec 28 09:17:16.614799506 (545h48m16.485882541s) [upload=478617]: Waiting for error group managing segments... Dec 28 09:17:16.903181063 (545h48m16.774264098s) [noise=true , address=145.79.227.18:4007 , piece_id=5IDHZHUS5ZEM7FBVMBEOBEXOWOHZGONRYAZ463USDUNK4KG3R6SQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.906044847 (545h48m16.777127892s) [noise=true , address=145.79.227.37:4567 , piece_id=6ROLV67SCGVXL2NROK74CJITQEXIAQLEAH2XKLB7SXUTOBTOR7FA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.910705191 (545h48m16.781788236s) [noise=true , address=145.79.227.14:4838 , piece_id=X2RHCXHDIFMA2ENAOMCBWH6VQIKVMOMZXIUTOQ3ABDEKSHDJMTYA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.917271434 (545h48m16.788354469s) [noise=true , address=145.79.227.45:4230 , piece_id=OMTIPMJPIGIYMVF3WRHITWOWFVYGGJ4MZUXTT4CJVIZ2P36Z34UA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.919809978 (545h48m16.790893013s) [noise=true , address=145.79.227.16:4911 , piece_id=DR6ZIDW3RZWHQ2LKJMRWGGFBYYNJ5J5EABFSWXMGWIDK3RCDQ42Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.920161106 (545h48m16.791244151s) [noise=true , address=145.79.227.48:4301 , piece_id=KNXU2A77GZ7SPGH6THLVCISROEFUI3NZ7XAZMRINBGMLOFIXXPHQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.923630453 (545h48m16.794713488s) [noise=true , address=145.79.227.27:4689 , piece_id=55Q7U6KFSNU67CX722SY5PNMDTH4ER7APDGH7BKAJGUP6NP3PQGA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.926877115 (545h48m16.79796016s) [noise=true , address=145.79.227.51:4409 , piece_id=ARCTECRUUY2U5FSUDM7OXC2CEXZJO6SNTEX2SNIFTMM2RXELUFLQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.927652300 (545h48m16.798735345s) [noise=true , address=145.79.227.29:4942 , piece_id=BEI5RIBBQZVGE3XSLPX266GUIQBE265DRP7NLV7ORZIJVIGDM54Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:16.932476477 (545h48m16.803559512s) [noise=true , address=145.79.227.47:4266 , piece_id=3B32OOXLO2XFPDFPUPWD63MFRQXOFAQMXNPKX6CC7KASVXJRE54A, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.137987024 (545h48m17.009070059s) [noise=true , address=45.91.116.5:4297 , piece_id=SVPRNPD3F43QH5BFVSWSS2N55USHC6KVXXGGXAQIBKJZAISC7HDA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.162012623 (545h48m17.033095658s) [noise=true , address=45.91.116.10:4081 , piece_id=TU3A5BSEQARMRWX7NNCXQ6DXLBU3MUTH6BEWMRN4VDEIM4Z75Y6Q, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.170451531 (545h48m17.041534566s) [noise=true , address=45.91.116.4:4940 , piece_id=EUK5DFYAMPOTZKKQ27QMJLFBYEC74IJ4ETU4XR2OKUHQPBWYRNOA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: <nil> Dec 28 09:17:17.170477551 (545h48m17.041560596s) [seg_pos={0 0}, upload=478617]: Segment reached optimal threshold of 13 pieces. Dec 28 09:17:17.170658676 (545h48m17.041741711s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 09:17:17.170665038 (545h48m17.041748083s) [noise=true , address=45.91.116.7:4012 , piece_id=AQSJYWX4YY5RLPOB6HCR27FNJ62CHAUPZTYFW7PJ5A7HCZTDJCYQ, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.170887782 (545h48m17.041970817s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 09:17:17.170893714 (545h48m17.041976759s) [noise=true , address=45.91.116.13:4156 , piece_id=26ZLTLHJYAVGM6AWNH674TEXHV6Q7WAXFW23PJF7WUPZQNJUFS6A, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.171022088 (545h48m17.042105133s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36158->77.90.31.13:4629: use of closed network connection Dec 28 09:17:17.171028090 (545h48m17.042111125s) [noise=true , address=77.90.31.13:4629 , piece_id=GEAZFUVGIDJFLVBYJAVKQ6A2PXAJRICS2HQL3IHAQXOW6HUVE5FA, seg_pos={0 0}, upload=478617]: Long tail cancellation due to optimal threshold Dec 28 09:17:17.171044641 (545h48m17.042127676s) [seg_pos={0 0}, upload=478617]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:17:17.171109604 (545h48m17.042192649s) [upload=478617]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:17:17.171113993 (545h48m17.042197038s) [upload=478617]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:17:17.460388621 (545h48m17.331471656s) [upload=478617]: Done uploading segments... Dec 28 09:17:17.460403749 (545h48m17.331486784s) [upload=478617]: Upload finished. err: <nil> Duration: 30.003140583s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XX.......X..X......XXXX.....XX.X.....X......X..............] duration: 5.921260915s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:16:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.921317033s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.XX....X....X.XX..XX..X........X.X...X....X.X.....X.....XXX.] duration: 3.731557948s trace: https://jaeger.dev.storj.io/trace/5f26bb3dcf8b2173 FAIL (2025-12-28 09:16:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.731577525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.XX.......X..X......XXXX.....XX.X.....X......X.............] duration: 5.738254117s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:16:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.738277242s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [X.XX....X....X.XX..XX..X........X.X...X....X.X.....X.....XXX] duration: 5.000897987s trace: https://jaeger.dev.storj.io/trace/5f26bb3dcf8b2173 FAIL (2025-12-28 09:15:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000923315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX.XX.......X..X......XXXX.....XX.X.....X......X............] duration: 6.208531396s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:15:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 6.208557204s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XX.XX....X....X.XX..XX..X........X.X...X....X.X.....X.....XX] duration: 5.000647921s trace: https://jaeger.dev.storj.io/trace/11571e3157eb5e5d FAIL (2025-12-28 09:14:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000671907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XXX.XX....X....X.XX..XX..X........X.X...X....X.X.....X.....X] duration: 5.001160444s trace: https://jaeger.dev.storj.io/trace/1657c223bab1b1c7 FAIL (2025-12-28 09:14:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001186556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX.XX.......X..X......XXXX.....XX.X.....X......X...........] duration: 5.931938216s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:13:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.931964655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.XX.XX.......X..X......XXXX.....XX.X.....X......X..........] duration: 5.945681686s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:13:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.945711934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.XX.XX.......X..X......XXXX.....XX.X.....X......X.........] duration: 5.6653475s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:13:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.665375805s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XXXX.XX....X....X.XX..XX..X........X.X...X....X.X.....X.....] duration: 3.562162782s trace: https://jaeger.dev.storj.io/trace/5749b6bdf69037d8 FAIL (2025-12-28 09:12:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.562186496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.XX.XX.......X..X......XXXX.....XX.X.....X......X........] duration: 5.956113208s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.956142865s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.XXXX.XX....X....X.XX..XX..X........X.X...X....X.X.....X....] duration: 3.793623291s trace: https://jaeger.dev.storj.io/trace/5749b6bdf69037d8 FAIL (2025-12-28 09:12:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.793652707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.XX.XX.......X..X......XXXX.....XX.X.....X......X.......] duration: 11.319090955s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:12:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 11.319118107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X...X.XX.XX.......X..X......XXXX.....XX.X.....X......X......] duration: 5.902579811s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:12:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.902612543s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..XXXX.XX....X....X.XX..XX..X........X.X...X....X.X.....X...] duration: 3.68707677s trace: https://jaeger.dev.storj.io/trace/5749b6bdf69037d8 FAIL (2025-12-28 09:11:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.68710369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X...X.XX.XX.......X..X......XXXX.....XX.X.....X......X.....] duration: 5.723107478s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:11:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.723137585s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...XXXX.XX....X....X.XX..XX..X........X.X...X....X.X.....X..] duration: 3.4596076s trace: https://jaeger.dev.storj.io/trace/5749b6bdf69037d8 FAIL (2025-12-28 09:10:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459731264s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [....XXXX.XX....X....X.XX..XX..X........X.X...X....X.X.....X.] duration: 3.529574807s trace: https://jaeger.dev.storj.io/trace/5749b6bdf69037d8 FAIL (2025-12-28 09:10:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.529598763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X...X.XX.XX.......X..X......XXXX.....XX.X.....X......X....] duration: 5.965888672s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:10:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.965918858s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X....XXXX.XX....X....X.XX..XX..X........X.X...X....X.X.....X] duration: 5.000766156s trace: https://jaeger.dev.storj.io/trace/5749b6bdf69037d8 FAIL (2025-12-28 09:09:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000790412s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X...X.XX.XX.......X..X......XXXX.....XX.X.....X......X...] duration: 5.906944631s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:09:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.907010977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X...X.XX.XX.......X..X......XXXX.....XX.X.....X......X..] duration: 5.986661087s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:08:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 5.986693289s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.X....XXXX.XX....X....X.XX..XX..X........X.X...X....X.X.....] duration: 3.43182351s trace: https://jaeger.dev.storj.io/trace/1c37b52926364294 FAIL (2025-12-28 09:08:39 UTC) orbiter warning: context deadline exceeded Duration: 3.431853519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X.X....XXXX.XX....X....X.XX..XX..X........X.X...X....X.X....] duration: 5.000477931s trace: https://jaeger.dev.storj.io/trace/1c37b52926364294 FAIL (2025-12-28 09:08:08 UTC) orbiter warning: context deadline exceeded Duration: 5.000610093s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....X...X.XX.XX.......X..X......XXXX.....XX.X.....X......X.] duration: 6.001326091s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:07:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 6.001355648s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [XX.X....XXXX.XX....X....X.XX..XX..X........X.X...X....X.X...] duration: 3.589514451s trace: https://jaeger.dev.storj.io/trace/1c37b52926364294 FAIL (2025-12-28 09:07:20 UTC) orbiter warning: context deadline exceeded Duration: 3.589538718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX....X...X.XX.XX.......X..X......XXXX.....XX.X.....X......X] duration: 30.003105084s trace: https://jaeger.dev.storj.io/trace/668044575496c959 FAIL (2025-12-28 09:07:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ) context deadline exceeded ======LOGS====== Dec 28 09:06:45.074767388 (545h37m44.945850423s) [upload=478459]: Starting upload Dec 28 09:06:45.074821361 (545h37m44.945904416s) [upload=478459]: Done starting upload Dec 28 09:06:45.074870825 (545h37m44.94595387s) [upload=478459]: Uploading segments... Dec 28 09:06:45.075067079 (545h37m44.946150124s) [upload=478459]: Got next segment. Inline: false Dec 28 09:06:45.075126030 (545h37m44.946209065s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:06:45.075130799 (545h37m44.946213834s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:06:45.320752036 (545h37m45.191835081s) [seg_pos={0 0}, upload=478459]: Begin upload segment... Dec 28 09:06:45.320804505 (545h37m45.19188754s) [seg_pos={0 0}, upload=478459]: Done begin upload segment. Dec 28 09:06:45.320831667 (545h37m45.191914702s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.320855342 (545h37m45.191938377s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321396933 (545h37m45.192479968s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.321770625 (545h37m45.19285366s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322288461 (545h37m45.193371506s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322650300 (545h37m45.193733335s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.322744940 (545h37m45.193827975s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323138139 (545h37m45.194221174s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.323527390 (545h37m45.194610425s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324265676 (545h37m45.195348721s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324596716 (545h37m45.195679761s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324627274 (545h37m45.19571031s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.324950910 (545h37m45.196033945s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325052855 (545h37m45.1961359s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325254789 (545h37m45.196337834s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:45.325489567 (545h37m45.196572612s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Uploading piece... Dec 28 09:06:47.028437236 (545h37m46.899520291s) [upload=478459]: Next returned nil segment Dec 28 09:06:47.028464899 (545h37m46.899547934s) [upload=478459]: Waiting for error group managing segments... Dec 28 09:06:47.316813305 (545h37m47.18789634s) [noise=true , address=145.79.227.27:4689 , piece_id=LO4GN3SBP7PVCGSGJJTY366P5JCPVDSXMR6RHM3IVAHAYTVI55NQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329090713 (545h37m47.200173758s) [noise=true , address=145.79.227.42:4121 , piece_id=HFWTDCATFYWAQOMXZ2VRGZO5SLBRCCWGRYGBB3PJHU2YI37BJ77A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329281776 (545h37m47.200364822s) [noise=true , address=145.79.227.17:4971 , piece_id=KZLDET6LEMSZUOIPSUYWUHDKPCYMEI6TJME4NFFBWJI6UQJ7ZRUQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.329545619 (545h37m47.200628664s) [noise=true , address=145.79.227.23:4547 , piece_id=LM4B34FWD3PR4UTRB7MXTPIUFQ4U4LSWIFKZR6N36U4P2INTTHYA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.332727538 (545h37m47.203810583s) [noise=true , address=145.79.227.32:4835 , piece_id=G55SUQHTOLHPPFUTH3AGOQ46HZZYMR5BM55KSXZODGYB246GGBMQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.335138007 (545h37m47.206221053s) [noise=true , address=145.79.227.16:4909 , piece_id=AH7DIY3V76LU66VJCI3V65NEAZZ6EZ6CNLUCH436QXPBWMBFTH2A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.336096563 (545h37m47.207179618s) [noise=true , address=145.79.227.36:4517 , piece_id=YA4IWLDQXSVUWDVZQJAQK7W4F7TVMJA3PCQTS66K3KQFIABX6P7Q, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.349342696 (545h37m47.220425731s) [noise=true , address=145.79.227.45:4229 , piece_id=FE2K7BZEFQ6THR2YPYEA2RUE7Y5MAV3YXSNNP7BXRNRF5L5SPZNQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.353314238 (545h37m47.224397273s) [noise=true , address=145.79.227.7:4293 , piece_id=A43EW6LSV4OOG4XCSOZKXGBHPLKIFD2CHGCSETSZ5AIX4JAXSKQQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.354577955 (545h37m47.225661s) [noise=true , address=145.79.227.2:4114 , piece_id=CYQQNA5EIBAETIUZ4CSL5Z7EI7HG76NZC5ETEPCZK4HPONCCJS3A, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.556318844 (545h37m47.427401889s) [noise=true , address=45.91.116.8:4045 , piece_id=7MMB6T4F73GDXTI6TA7DF3LQ7VA3FDDNPQJCODWIN3RP6QSTKPFA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.565472194 (545h37m47.436555229s) [noise=true , address=45.91.116.6:4975 , piece_id=Y2HNELCNNCXU4EGUOA524RPPHBEGK776VLWGUTRXBX4R5X7KIYKQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587710570 (545h37m47.458793605s) [noise=true , address=45.91.116.14:4190 , piece_id=2JO3VXBOWMGLFDBCYQW57CAIL5J2F23NPGSUWHPJTK5CSH5KSGPQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: <nil> Dec 28 09:06:47.587749894 (545h37m47.458832939s) [seg_pos={0 0}, upload=478459]: Segment reached optimal threshold of 13 pieces. Dec 28 09:06:47.587940758 (545h37m47.459023803s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.587946730 (545h37m47.459029765s) [noise=true , address=45.91.116.11:4118 , piece_id=AGZDFM3WEKQ6CBKAE4ELL3NEFWUNFIQYX3R54DY4FL57WFSHBUGA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588091796 (545h37m47.459174831s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588098598 (545h37m47.459181643s) [noise=true , address=145.79.227.19:4079 , piece_id=FCNRNOP4U5T2JSP3QJXUJ7F25ZTDQR2ZTITMOWZE2R5V4AVU7HQA, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588236491 (545h37m47.459319546s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 09:06:47.588255107 (545h37m47.459338152s) [noise=true , address=45.91.116.10:4081 , piece_id=SFZDDQQE32YZJFCXQBGVF5SUNS7XZVGSYLRUEOPBWU3TPQEXSLIQ, seg_pos={0 0}, upload=478459]: Long tail cancellation due to optimal threshold Dec 28 09:06:47.588282539 (545h37m47.459365584s) [seg_pos={0 0}, upload=478459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:06:47.588405433 (545h37m47.459488478s) [upload=478459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:06:47.588412115 (545h37m47.45949516s) [upload=478459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:06:47.881589988 (545h37m47.752673033s) [upload=478459]: Done uploading segments... Dec 28 09:06:47.881617691 (545h37m47.752700726s) [upload=478459]: Upload finished. err: <nil> Duration: 30.003120243s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.XX.X....XXXX.XX....X....X.XX..XX..X........X.X...X....X.X..] duration: 4.462115896s trace: https://jaeger.dev.storj.io/trace/1c37b52926364294 FAIL (2025-12-28 09:06:56 UTC) orbiter warning: context deadline exceeded Duration: 4.462138089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX....X...X.XX.XX.......X..X......XXXX.....XX.X.....X......] duration: 5.900912445s trace: https://jaeger.dev.storj.io/trace/3c2199157916627a FAIL (2025-12-28 09:06:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: NYBKXBI3BJK5IKY3AAVG2AHPAJ5HNMBOO3ZZMXIY33TEXDF7YJKA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: IT4XJWM4PLQXP56SGYQBC5TTVRMB242DMS5VXSMXRB4ER4DKWDJA) context deadline exceeded ======LOGS====== 09:02:15.935625974 (545h33m15.806709009s) [upload=478395]: Starting upload Dec 28 09:02:15.935701268 (545h33m15.806784323s) [upload=478395]: Done starting upload Dec 28 09:02:15.935951494 (545h33m15.80703454s) [upload=478395]: Uploading segments... Dec 28 09:02:15.936098294 (545h33m15.807181329s) [upload=478395]: Got next segment. Inline: false Dec 28 09:02:15.936151434 (545h33m15.80723448s) [upload=478395]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:02:15.936158068 (545h33m15.807241103s) [upload=478395]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:02:16.179185073 (545h33m16.050268108s) [seg_pos={0 0}, upload=478395]: Begin upload segment... Dec 28 09:02:16.179244045 (545h33m16.050327091s) [seg_pos={0 0}, upload=478395]: Done begin upload segment. Dec 28 09:02:16.179288630 (545h33m16.050371685s) [noise=true , address=77.90.31.14:4663 , piece_id=KOWC6XL5EC3D2W7EK4RM5JMH6UJCAEOMKXV5WMQF3L3VHTCOZLMA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.179438196 (545h33m16.050521231s) [noise=true , address=77.90.31.17:4773 , piece_id=BMMPOJ7ABULBU6RDAXKJRX4KA7E2SR3P6FHB7GODPJIW5MCSIOWQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180012469 (545h33m16.051095514s) [noise=true , address=145.79.227.14:4839 , piece_id=7HVNCH6HIBVM3SSM3C244A2MXBYYJP7RG5KUEHFTWOTEFA4Q5FUQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180335464 (545h33m16.051418499s) [noise=true , address=145.79.227.32:4836 , piece_id=76C6AHTLOTP3ZLNAOTNI6KCPU3Q5VSYJ7ZPRVNK7I3BIMYUQI6HA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180606229 (545h33m16.051689274s) [noise=true , address=145.79.227.42:4121 , piece_id=RPIELQK5BE37LFUVHHWTVBVKPK2ACHQEN6AP6H65MISKCCPHZWXQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180622100 (545h33m16.051705135s) [noise=true , address=45.91.116.12:4227 , piece_id=QKHI4DMLXTFYQEOT6DR7PWOIM5BW47APM22PLOPJDDSIYUCDRIYA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180886002 (545h33m16.051969047s) [noise=true , address=77.90.31.16:4736 , piece_id=2GUJLGQGK6YWEB6VBGHTZ2QAZIECWNUZPMUVPYVEQF5P4XB4WVOQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.181151628 (545h33m16.052234673s) [noise=true , address=45.91.116.7:4011 , piece_id=K4TN54SKSZHCUTQTGUZUPDXAOVTIKSV3INTWC3AZIFMCMBC56IRQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.181666859 (545h33m16.052749904s) [noise=true , address=45.91.116.10:4083 , piece_id=NYBKXBI3BJK5IKY3AAVG2AHPAJ5HNMBOO3ZZMXIY33TEXDF7YJKA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.182299083 (545h33m16.053382128s) [noise=true , address=45.91.116.8:4048 , piece_id=JGMM4U5EI77S2BTFLTOSRXFJQ4TVLM4HLOM45FD2TIQRUHL4TYLA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.183318654 (545h33m16.054401699s) [noise=true , address=45.91.116.14:4191 , piece_id=IT4XJWM4PLQXP56SGYQBC5TTVRMB242DMS5VXSMXRB4ER4DKWDJA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.183799330 (545h33m16.054882365s) [noise=true , address=145.79.227.9:4365 , piece_id=H6XSSXZBBGSDSXBRJW5DPUOBCMT5GVGFWZRW7QDGICHICZEOYYRA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.184287559 (545h33m16.055370624s) [noise=true , address=145.79.227.45:4231 , piece_id=KQNDRZNF2S7PHHLKT234ZAO67XRG4RC23ZOYWSP7ZPCM26YWVFZQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.184680337 (545h33m16.055763372s) [noise=true , address=145.79.227.7:4293 , piece_id=Q3I4NQKFXZXBHW4YG736EF2XLKR2FFABGOTWOFPA5IFZJ73R2VWQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.185365462 (545h33m16.056448507s) [noise=true , address=45.91.116.6:4976 , piece_id=IANPWH25IHDHQ2KR7A5VKH7VQKOF5BX2NB2D7MACDLEY6WAOIQDQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.186231211 (545h33m16.057314256s) [noise=true , address=145.79.227.50:4375 , piece_id=RBD7K3AKX5QXXKOAPROM3ZSLE7ZWHZFPTSP3JGY57D6VBMWFGDBA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:17.816555893 (545h33m17.687638928s) [upload=478395]: Next returned nil segment Dec 28 09:02:17.816570581 (545h33m17.687653626s) [upload=478395]: Waiting for error group managing segments... Dec 28 09:02:18.189517868 (545h33m18.060600903s) [noise=true , address=145.79.227.7:4293 , piece_id=Q3I4NQKFXZXBHW4YG736EF2XLKR2FFABGOTWOFPA5IFZJ73R2VWQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.194330282 (545h33m18.065413317s) [noise=true , address=145.79.227.9:4365 , piece_id=H6XSSXZBBGSDSXBRJW5DPUOBCMT5GVGFWZRW7QDGICHICZEOYYRA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.194477542 (545h33m18.065560587s) [noise=true , address=145.79.227.50:4375 , piece_id=RBD7K3AKX5QXXKOAPROM3ZSLE7ZWHZFPTSP3JGY57D6VBMWFGDBA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.206666572 (545h33m18.077749617s) [noise=true , address=145.79.227.14:4839 , piece_id=7HVNCH6HIBVM3SSM3C244A2MXBYYJP7RG5KUEHFTWOTEFA4Q5FUQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.207299126 (545h33m18.078382161s) [noise=true , address=145.79.227.32:4836 , piece_id=76C6AHTLOTP3ZLNAOTNI6KCPU3Q5VSYJ7ZPRVNK7I3BIMYUQI6HA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.211166942 (545h33m18.082250047s) [noise=true , address=145.79.227.45:4231 , piece_id=KQNDRZNF2S7PHHLKT234ZAO67XRG4RC23ZOYWSP7ZPCM26YWVFZQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.222593901 (545h33m18.093676936s) [noise=true , address=145.79.227.42:4121 , piece_id=RPIELQK5BE37LFUVHHWTVBVKPK2ACHQEN6AP6H65MISKCCPHZWXQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.434233279 (545h33m18.305316314s) [noise=true , address=45.91.116.10:4083 , piece_id=NYBKXBI3BJK5IKY3AAVG2AHPAJ5HNMBOO3ZZMXIY33TEXDF7YJKA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.488831443 (545h33m18.359914488s) [noise=true , address=77.90.31.14:4663 , piece_id=KOWC6XL5EC3D2W7EK4RM5JMH6UJCAEOMKXV5WMQF3L3VHTCOZLMA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.526343533 (545h33m18.397426608s) [noise=true , address=77.90.31.17:4773 , piece_id=BMMPOJ7ABULBU6RDAXKJRX4KA7E2SR3P6FHB7GODPJIW5MCSIOWQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.529689093 (545h33m18.400772128s) [noise=true , address=45.91.116.12:4227 , piece_id=QKHI4DMLXTFYQEOT6DR7PWOIM5BW47APM22PLOPJDDSIYUCDRIYA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.567241489 (545h33m18.438324524s) [noise=true , address=45.91.116.7:4011 , piece_id=K4TN54SKSZHCUTQTGUZUPDXAOVTIKSV3INTWC3AZIFMCMBC56IRQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.669622681 (545h33m18.540705726s) [noise=true , address=45.91.116.14:4191 , piece_id=IT4XJWM4PLQXP56SGYQBC5TTVRMB242DMS5VXSMXRB4ER4DKWDJA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.669652398 (545h33m18.540735443s) [seg_pos={0 0}, upload=478395]: Segment reached optimal threshold of 13 pieces. Dec 28 09:02:18.669927552 (545h33m18.541010607s) [noise=true , address=45.91.116.6:4976 , piece_id=IANPWH25IHDHQ2KR7A5VKH7VQKOF5BX2NB2D7MACDLEY6WAOIQDQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 09:02:18.669943232 (545h33m18.541026277s) [noise=true , address=45.91.116.6:4976 , piece_id=IANPWH25IHDHQ2KR7A5VKH7VQKOF5BX2NB2D7MACDLEY6WAOIQDQ, seg_pos={0 0}, upload=478395]: Long tail cancellation due to optimal threshold Dec 28 09:02:18.670218736 (545h33m18.541301781s) [noise=true , address=77.90.31.16:4736 , piece_id=2GUJLGQGK6YWEB6VBGHTZ2QAZIECWNUZPMUVPYVEQF5P4XB4WVOQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 09:02:18.670231961 (545h33m18.541315007s) [noise=true , address=77.90.31.16:4736 , piece_id=2GUJLGQGK6YWEB6VBGHTZ2QAZIECWNUZPMUVPYVEQF5P4XB4WVOQ, seg_pos={0 0}, upload=478395]: Long tail cancellation due to optimal threshold Dec 28 09:02:18.670258663 (545h33m18.541341718s) [noise=true , address=45.91.116.8:4048 , piece_id=JGMM4U5EI77S2BTFLTOSRXFJQ4TVLM4HLOM45FD2TIQRUHL4TYLA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:58658->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:58658->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58658->45.91.116.8:4048: use of closed network connection Dec 28 09:02:18.670287767 (545h33m18.541370812s) [noise=true , address=45.91.116.8:4048 , piece_id=JGMM4U5EI77S2BTFLTOSRXFJQ4TVLM4HLOM45FD2TIQRUHL4TYLA, seg_pos={0 0}, upload=478395]: Long tail cancellation due to optimal threshold Dec 28 09:02:18.670329828 (545h33m18.541412873s) [seg_pos={0 0}, upload=478395]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:02:18.670473381 (545h33m18.541556416s) [upload=478395]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:02:18.670481126 (545h33m18.541564181s) [upload=478395]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:02:18.962569312 (545h33m18.833652347s) [upload=478395]: Done uploading segments... Dec 28 09:02:18.962592056 (545h33m18.833675091s) [upload=478395]: Upload finished. err: <nil> Duration: 5.900941461s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [..XX.X....XXXX.XX....X....X.XX..XX..X........X.X...X....X.X.] duration: 3.540876066s trace: https://jaeger.dev.storj.io/trace/1c37b52926364294 FAIL (2025-12-28 09:06:03 UTC) orbiter warning: context deadline exceeded Duration: 3.540902526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [...XX.X....XXXX.XX....X....X.XX..XX..X........X.X...X....X.X] duration: 5.000632897s trace: https://jaeger.dev.storj.io/trace/1c37b52926364294 FAIL (2025-12-28 09:05:50 UTC) orbiter warning: context deadline exceeded Duration: 5.000656993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX....X...X.XX.XX.......X..X......XXXX.....XX.X.....X.....] duration: 5.843852624s trace: https://jaeger.dev.storj.io/trace/3c2199157916627a FAIL (2025-12-28 09:05:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: NYBKXBI3BJK5IKY3AAVG2AHPAJ5HNMBOO3ZZMXIY33TEXDF7YJKA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: IT4XJWM4PLQXP56SGYQBC5TTVRMB242DMS5VXSMXRB4ER4DKWDJA) context deadline exceeded ======LOGS====== 09:02:15.935625974 (545h33m15.806709009s) [upload=478395]: Starting upload Dec 28 09:02:15.935701268 (545h33m15.806784323s) [upload=478395]: Done starting upload Dec 28 09:02:15.935951494 (545h33m15.80703454s) [upload=478395]: Uploading segments... Dec 28 09:02:15.936098294 (545h33m15.807181329s) [upload=478395]: Got next segment. Inline: false Dec 28 09:02:15.936151434 (545h33m15.80723448s) [upload=478395]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:02:15.936158068 (545h33m15.807241103s) [upload=478395]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:02:16.179185073 (545h33m16.050268108s) [seg_pos={0 0}, upload=478395]: Begin upload segment... Dec 28 09:02:16.179244045 (545h33m16.050327091s) [seg_pos={0 0}, upload=478395]: Done begin upload segment. Dec 28 09:02:16.179288630 (545h33m16.050371685s) [noise=true , address=77.90.31.14:4663 , piece_id=KOWC6XL5EC3D2W7EK4RM5JMH6UJCAEOMKXV5WMQF3L3VHTCOZLMA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.179438196 (545h33m16.050521231s) [noise=true , address=77.90.31.17:4773 , piece_id=BMMPOJ7ABULBU6RDAXKJRX4KA7E2SR3P6FHB7GODPJIW5MCSIOWQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180012469 (545h33m16.051095514s) [noise=true , address=145.79.227.14:4839 , piece_id=7HVNCH6HIBVM3SSM3C244A2MXBYYJP7RG5KUEHFTWOTEFA4Q5FUQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180335464 (545h33m16.051418499s) [noise=true , address=145.79.227.32:4836 , piece_id=76C6AHTLOTP3ZLNAOTNI6KCPU3Q5VSYJ7ZPRVNK7I3BIMYUQI6HA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180606229 (545h33m16.051689274s) [noise=true , address=145.79.227.42:4121 , piece_id=RPIELQK5BE37LFUVHHWTVBVKPK2ACHQEN6AP6H65MISKCCPHZWXQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180622100 (545h33m16.051705135s) [noise=true , address=45.91.116.12:4227 , piece_id=QKHI4DMLXTFYQEOT6DR7PWOIM5BW47APM22PLOPJDDSIYUCDRIYA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180886002 (545h33m16.051969047s) [noise=true , address=77.90.31.16:4736 , piece_id=2GUJLGQGK6YWEB6VBGHTZ2QAZIECWNUZPMUVPYVEQF5P4XB4WVOQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.181151628 (545h33m16.052234673s) [noise=true , address=45.91.116.7:4011 , piece_id=K4TN54SKSZHCUTQTGUZUPDXAOVTIKSV3INTWC3AZIFMCMBC56IRQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.181666859 (545h33m16.052749904s) [noise=true , address=45.91.116.10:4083 , piece_id=NYBKXBI3BJK5IKY3AAVG2AHPAJ5HNMBOO3ZZMXIY33TEXDF7YJKA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.182299083 (545h33m16.053382128s) [noise=true , address=45.91.116.8:4048 , piece_id=JGMM4U5EI77S2BTFLTOSRXFJQ4TVLM4HLOM45FD2TIQRUHL4TYLA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.183318654 (545h33m16.054401699s) [noise=true , address=45.91.116.14:4191 , piece_id=IT4XJWM4PLQXP56SGYQBC5TTVRMB242DMS5VXSMXRB4ER4DKWDJA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.183799330 (545h33m16.054882365s) [noise=true , address=145.79.227.9:4365 , piece_id=H6XSSXZBBGSDSXBRJW5DPUOBCMT5GVGFWZRW7QDGICHICZEOYYRA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.184287559 (545h33m16.055370624s) [noise=true , address=145.79.227.45:4231 , piece_id=KQNDRZNF2S7PHHLKT234ZAO67XRG4RC23ZOYWSP7ZPCM26YWVFZQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.184680337 (545h33m16.055763372s) [noise=true , address=145.79.227.7:4293 , piece_id=Q3I4NQKFXZXBHW4YG736EF2XLKR2FFABGOTWOFPA5IFZJ73R2VWQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.185365462 (545h33m16.056448507s) [noise=true , address=45.91.116.6:4976 , piece_id=IANPWH25IHDHQ2KR7A5VKH7VQKOF5BX2NB2D7MACDLEY6WAOIQDQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.186231211 (545h33m16.057314256s) [noise=true , address=145.79.227.50:4375 , piece_id=RBD7K3AKX5QXXKOAPROM3ZSLE7ZWHZFPTSP3JGY57D6VBMWFGDBA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:17.816555893 (545h33m17.687638928s) [upload=478395]: Next returned nil segment Dec 28 09:02:17.816570581 (545h33m17.687653626s) [upload=478395]: Waiting for error group managing segments... Dec 28 09:02:18.189517868 (545h33m18.060600903s) [noise=true , address=145.79.227.7:4293 , piece_id=Q3I4NQKFXZXBHW4YG736EF2XLKR2FFABGOTWOFPA5IFZJ73R2VWQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.194330282 (545h33m18.065413317s) [noise=true , address=145.79.227.9:4365 , piece_id=H6XSSXZBBGSDSXBRJW5DPUOBCMT5GVGFWZRW7QDGICHICZEOYYRA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.194477542 (545h33m18.065560587s) [noise=true , address=145.79.227.50:4375 , piece_id=RBD7K3AKX5QXXKOAPROM3ZSLE7ZWHZFPTSP3JGY57D6VBMWFGDBA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.206666572 (545h33m18.077749617s) [noise=true , address=145.79.227.14:4839 , piece_id=7HVNCH6HIBVM3SSM3C244A2MXBYYJP7RG5KUEHFTWOTEFA4Q5FUQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.207299126 (545h33m18.078382161s) [noise=true , address=145.79.227.32:4836 , piece_id=76C6AHTLOTP3ZLNAOTNI6KCPU3Q5VSYJ7ZPRVNK7I3BIMYUQI6HA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.211166942 (545h33m18.082250047s) [noise=true , address=145.79.227.45:4231 , piece_id=KQNDRZNF2S7PHHLKT234ZAO67XRG4RC23ZOYWSP7ZPCM26YWVFZQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.222593901 (545h33m18.093676936s) [noise=true , address=145.79.227.42:4121 , piece_id=RPIELQK5BE37LFUVHHWTVBVKPK2ACHQEN6AP6H65MISKCCPHZWXQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.434233279 (545h33m18.305316314s) [noise=true , address=45.91.116.10:4083 , piece_id=NYBKXBI3BJK5IKY3AAVG2AHPAJ5HNMBOO3ZZMXIY33TEXDF7YJKA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.488831443 (545h33m18.359914488s) [noise=true , address=77.90.31.14:4663 , piece_id=KOWC6XL5EC3D2W7EK4RM5JMH6UJCAEOMKXV5WMQF3L3VHTCOZLMA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.526343533 (545h33m18.397426608s) [noise=true , address=77.90.31.17:4773 , piece_id=BMMPOJ7ABULBU6RDAXKJRX4KA7E2SR3P6FHB7GODPJIW5MCSIOWQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.529689093 (545h33m18.400772128s) [noise=true , address=45.91.116.12:4227 , piece_id=QKHI4DMLXTFYQEOT6DR7PWOIM5BW47APM22PLOPJDDSIYUCDRIYA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.567241489 (545h33m18.438324524s) [noise=true , address=45.91.116.7:4011 , piece_id=K4TN54SKSZHCUTQTGUZUPDXAOVTIKSV3INTWC3AZIFMCMBC56IRQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.669622681 (545h33m18.540705726s) [noise=true , address=45.91.116.14:4191 , piece_id=IT4XJWM4PLQXP56SGYQBC5TTVRMB242DMS5VXSMXRB4ER4DKWDJA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.669652398 (545h33m18.540735443s) [seg_pos={0 0}, upload=478395]: Segment reached optimal threshold of 13 pieces. Dec 28 09:02:18.669927552 (545h33m18.541010607s) [noise=true , address=45.91.116.6:4976 , piece_id=IANPWH25IHDHQ2KR7A5VKH7VQKOF5BX2NB2D7MACDLEY6WAOIQDQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 09:02:18.669943232 (545h33m18.541026277s) [noise=true , address=45.91.116.6:4976 , piece_id=IANPWH25IHDHQ2KR7A5VKH7VQKOF5BX2NB2D7MACDLEY6WAOIQDQ, seg_pos={0 0}, upload=478395]: Long tail cancellation due to optimal threshold Dec 28 09:02:18.670218736 (545h33m18.541301781s) [noise=true , address=77.90.31.16:4736 , piece_id=2GUJLGQGK6YWEB6VBGHTZ2QAZIECWNUZPMUVPYVEQF5P4XB4WVOQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 09:02:18.670231961 (545h33m18.541315007s) [noise=true , address=77.90.31.16:4736 , piece_id=2GUJLGQGK6YWEB6VBGHTZ2QAZIECWNUZPMUVPYVEQF5P4XB4WVOQ, seg_pos={0 0}, upload=478395]: Long tail cancellation due to optimal threshold Dec 28 09:02:18.670258663 (545h33m18.541341718s) [noise=true , address=45.91.116.8:4048 , piece_id=JGMM4U5EI77S2BTFLTOSRXFJQ4TVLM4HLOM45FD2TIQRUHL4TYLA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:58658->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:58658->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58658->45.91.116.8:4048: use of closed network connection Dec 28 09:02:18.670287767 (545h33m18.541370812s) [noise=true , address=45.91.116.8:4048 , piece_id=JGMM4U5EI77S2BTFLTOSRXFJQ4TVLM4HLOM45FD2TIQRUHL4TYLA, seg_pos={0 0}, upload=478395]: Long tail cancellation due to optimal threshold Dec 28 09:02:18.670329828 (545h33m18.541412873s) [seg_pos={0 0}, upload=478395]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:02:18.670473381 (545h33m18.541556416s) [upload=478395]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:02:18.670481126 (545h33m18.541564181s) [upload=478395]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:02:18.962569312 (545h33m18.833652347s) [upload=478395]: Done uploading segments... Dec 28 09:02:18.962592056 (545h33m18.833675091s) [upload=478395]: Upload finished. err: <nil> Duration: 5.843882772s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [....XX.X....XXXX.XX....X....X.XX..XX..X........X.X...X....X.] duration: 4.963222101s trace: https://jaeger.dev.storj.io/trace/531fdc15dab2edc8 FAIL (2025-12-28 09:05:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.963257809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..XX....X...X.XX.XX.......X..X......XXXX.....XX.X.....X....] duration: 5.715002511s trace: https://jaeger.dev.storj.io/trace/3c2199157916627a FAIL (2025-12-28 09:04:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: NYBKXBI3BJK5IKY3AAVG2AHPAJ5HNMBOO3ZZMXIY33TEXDF7YJKA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: IT4XJWM4PLQXP56SGYQBC5TTVRMB242DMS5VXSMXRB4ER4DKWDJA) context deadline exceeded ======LOGS====== 09:02:15.935625974 (545h33m15.806709009s) [upload=478395]: Starting upload Dec 28 09:02:15.935701268 (545h33m15.806784323s) [upload=478395]: Done starting upload Dec 28 09:02:15.935951494 (545h33m15.80703454s) [upload=478395]: Uploading segments... Dec 28 09:02:15.936098294 (545h33m15.807181329s) [upload=478395]: Got next segment. Inline: false Dec 28 09:02:15.936151434 (545h33m15.80723448s) [upload=478395]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:02:15.936158068 (545h33m15.807241103s) [upload=478395]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:02:16.179185073 (545h33m16.050268108s) [seg_pos={0 0}, upload=478395]: Begin upload segment... Dec 28 09:02:16.179244045 (545h33m16.050327091s) [seg_pos={0 0}, upload=478395]: Done begin upload segment. Dec 28 09:02:16.179288630 (545h33m16.050371685s) [noise=true , address=77.90.31.14:4663 , piece_id=KOWC6XL5EC3D2W7EK4RM5JMH6UJCAEOMKXV5WMQF3L3VHTCOZLMA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.179438196 (545h33m16.050521231s) [noise=true , address=77.90.31.17:4773 , piece_id=BMMPOJ7ABULBU6RDAXKJRX4KA7E2SR3P6FHB7GODPJIW5MCSIOWQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180012469 (545h33m16.051095514s) [noise=true , address=145.79.227.14:4839 , piece_id=7HVNCH6HIBVM3SSM3C244A2MXBYYJP7RG5KUEHFTWOTEFA4Q5FUQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180335464 (545h33m16.051418499s) [noise=true , address=145.79.227.32:4836 , piece_id=76C6AHTLOTP3ZLNAOTNI6KCPU3Q5VSYJ7ZPRVNK7I3BIMYUQI6HA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180606229 (545h33m16.051689274s) [noise=true , address=145.79.227.42:4121 , piece_id=RPIELQK5BE37LFUVHHWTVBVKPK2ACHQEN6AP6H65MISKCCPHZWXQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180622100 (545h33m16.051705135s) [noise=true , address=45.91.116.12:4227 , piece_id=QKHI4DMLXTFYQEOT6DR7PWOIM5BW47APM22PLOPJDDSIYUCDRIYA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.180886002 (545h33m16.051969047s) [noise=true , address=77.90.31.16:4736 , piece_id=2GUJLGQGK6YWEB6VBGHTZ2QAZIECWNUZPMUVPYVEQF5P4XB4WVOQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.181151628 (545h33m16.052234673s) [noise=true , address=45.91.116.7:4011 , piece_id=K4TN54SKSZHCUTQTGUZUPDXAOVTIKSV3INTWC3AZIFMCMBC56IRQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.181666859 (545h33m16.052749904s) [noise=true , address=45.91.116.10:4083 , piece_id=NYBKXBI3BJK5IKY3AAVG2AHPAJ5HNMBOO3ZZMXIY33TEXDF7YJKA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.182299083 (545h33m16.053382128s) [noise=true , address=45.91.116.8:4048 , piece_id=JGMM4U5EI77S2BTFLTOSRXFJQ4TVLM4HLOM45FD2TIQRUHL4TYLA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.183318654 (545h33m16.054401699s) [noise=true , address=45.91.116.14:4191 , piece_id=IT4XJWM4PLQXP56SGYQBC5TTVRMB242DMS5VXSMXRB4ER4DKWDJA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.183799330 (545h33m16.054882365s) [noise=true , address=145.79.227.9:4365 , piece_id=H6XSSXZBBGSDSXBRJW5DPUOBCMT5GVGFWZRW7QDGICHICZEOYYRA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.184287559 (545h33m16.055370624s) [noise=true , address=145.79.227.45:4231 , piece_id=KQNDRZNF2S7PHHLKT234ZAO67XRG4RC23ZOYWSP7ZPCM26YWVFZQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.184680337 (545h33m16.055763372s) [noise=true , address=145.79.227.7:4293 , piece_id=Q3I4NQKFXZXBHW4YG736EF2XLKR2FFABGOTWOFPA5IFZJ73R2VWQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.185365462 (545h33m16.056448507s) [noise=true , address=45.91.116.6:4976 , piece_id=IANPWH25IHDHQ2KR7A5VKH7VQKOF5BX2NB2D7MACDLEY6WAOIQDQ, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:16.186231211 (545h33m16.057314256s) [noise=true , address=145.79.227.50:4375 , piece_id=RBD7K3AKX5QXXKOAPROM3ZSLE7ZWHZFPTSP3JGY57D6VBMWFGDBA, seg_pos={0 0}, upload=478395]: Uploading piece... Dec 28 09:02:17.816555893 (545h33m17.687638928s) [upload=478395]: Next returned nil segment Dec 28 09:02:17.816570581 (545h33m17.687653626s) [upload=478395]: Waiting for error group managing segments... Dec 28 09:02:18.189517868 (545h33m18.060600903s) [noise=true , address=145.79.227.7:4293 , piece_id=Q3I4NQKFXZXBHW4YG736EF2XLKR2FFABGOTWOFPA5IFZJ73R2VWQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.194330282 (545h33m18.065413317s) [noise=true , address=145.79.227.9:4365 , piece_id=H6XSSXZBBGSDSXBRJW5DPUOBCMT5GVGFWZRW7QDGICHICZEOYYRA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.194477542 (545h33m18.065560587s) [noise=true , address=145.79.227.50:4375 , piece_id=RBD7K3AKX5QXXKOAPROM3ZSLE7ZWHZFPTSP3JGY57D6VBMWFGDBA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.206666572 (545h33m18.077749617s) [noise=true , address=145.79.227.14:4839 , piece_id=7HVNCH6HIBVM3SSM3C244A2MXBYYJP7RG5KUEHFTWOTEFA4Q5FUQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.207299126 (545h33m18.078382161s) [noise=true , address=145.79.227.32:4836 , piece_id=76C6AHTLOTP3ZLNAOTNI6KCPU3Q5VSYJ7ZPRVNK7I3BIMYUQI6HA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.211166942 (545h33m18.082250047s) [noise=true , address=145.79.227.45:4231 , piece_id=KQNDRZNF2S7PHHLKT234ZAO67XRG4RC23ZOYWSP7ZPCM26YWVFZQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.222593901 (545h33m18.093676936s) [noise=true , address=145.79.227.42:4121 , piece_id=RPIELQK5BE37LFUVHHWTVBVKPK2ACHQEN6AP6H65MISKCCPHZWXQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.434233279 (545h33m18.305316314s) [noise=true , address=45.91.116.10:4083 , piece_id=NYBKXBI3BJK5IKY3AAVG2AHPAJ5HNMBOO3ZZMXIY33TEXDF7YJKA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.488831443 (545h33m18.359914488s) [noise=true , address=77.90.31.14:4663 , piece_id=KOWC6XL5EC3D2W7EK4RM5JMH6UJCAEOMKXV5WMQF3L3VHTCOZLMA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.526343533 (545h33m18.397426608s) [noise=true , address=77.90.31.17:4773 , piece_id=BMMPOJ7ABULBU6RDAXKJRX4KA7E2SR3P6FHB7GODPJIW5MCSIOWQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.529689093 (545h33m18.400772128s) [noise=true , address=45.91.116.12:4227 , piece_id=QKHI4DMLXTFYQEOT6DR7PWOIM5BW47APM22PLOPJDDSIYUCDRIYA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.567241489 (545h33m18.438324524s) [noise=true , address=45.91.116.7:4011 , piece_id=K4TN54SKSZHCUTQTGUZUPDXAOVTIKSV3INTWC3AZIFMCMBC56IRQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.669622681 (545h33m18.540705726s) [noise=true , address=45.91.116.14:4191 , piece_id=IT4XJWM4PLQXP56SGYQBC5TTVRMB242DMS5VXSMXRB4ER4DKWDJA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: <nil> Dec 28 09:02:18.669652398 (545h33m18.540735443s) [seg_pos={0 0}, upload=478395]: Segment reached optimal threshold of 13 pieces. Dec 28 09:02:18.669927552 (545h33m18.541010607s) [noise=true , address=45.91.116.6:4976 , piece_id=IANPWH25IHDHQ2KR7A5VKH7VQKOF5BX2NB2D7MACDLEY6WAOIQDQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 09:02:18.669943232 (545h33m18.541026277s) [noise=true , address=45.91.116.6:4976 , piece_id=IANPWH25IHDHQ2KR7A5VKH7VQKOF5BX2NB2D7MACDLEY6WAOIQDQ, seg_pos={0 0}, upload=478395]: Long tail cancellation due to optimal threshold Dec 28 09:02:18.670218736 (545h33m18.541301781s) [noise=true , address=77.90.31.16:4736 , piece_id=2GUJLGQGK6YWEB6VBGHTZ2QAZIECWNUZPMUVPYVEQF5P4XB4WVOQ, seg_pos={0 0}, upload=478395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 09:02:18.670231961 (545h33m18.541315007s) [noise=true , address=77.90.31.16:4736 , piece_id=2GUJLGQGK6YWEB6VBGHTZ2QAZIECWNUZPMUVPYVEQF5P4XB4WVOQ, seg_pos={0 0}, upload=478395]: Long tail cancellation due to optimal threshold Dec 28 09:02:18.670258663 (545h33m18.541341718s) [noise=true , address=45.91.116.8:4048 , piece_id=JGMM4U5EI77S2BTFLTOSRXFJQ4TVLM4HLOM45FD2TIQRUHL4TYLA, seg_pos={0 0}, upload=478395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:58658->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:58658->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58658->45.91.116.8:4048: use of closed network connection Dec 28 09:02:18.670287767 (545h33m18.541370812s) [noise=true , address=45.91.116.8:4048 , piece_id=JGMM4U5EI77S2BTFLTOSRXFJQ4TVLM4HLOM45FD2TIQRUHL4TYLA, seg_pos={0 0}, upload=478395]: Long tail cancellation due to optimal threshold Dec 28 09:02:18.670329828 (545h33m18.541412873s) [seg_pos={0 0}, upload=478395]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:02:18.670473381 (545h33m18.541556416s) [upload=478395]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:02:18.670481126 (545h33m18.541564181s) [upload=478395]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:02:18.962569312 (545h33m18.833652347s) [upload=478395]: Done uploading segments... Dec 28 09:02:18.962592056 (545h33m18.833675091s) [upload=478395]: Upload finished. err: <nil> Duration: 5.715035104s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.....XX.X....XXXX.XX....X....X.XX..XX..X........X.X...X....X] duration: 5.001667617s trace: https://jaeger.dev.storj.io/trace/531fdc15dab2edc8 FAIL (2025-12-28 09:04:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001694268s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)