Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.XX.X..X..X.] duration: 3.788272224s trace: https://jaeger.dev.storj.io/trace/14cc117c56cd9868 FAIL (2025-12-16 13:09:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.788299777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.XX.X..X..X] duration: 5.002066905s trace: https://jaeger.dev.storj.io/trace/14cc117c56cd9868 FAIL (2025-12-16 13:09:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002093175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X..X..........................X.....X..] duration: 6.018527251s trace: https://jaeger.dev.storj.io/trace/5305a9e5a4c67493 FAIL (2025-12-16 13:09:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: AJCQN3JS34CN7NTABZFI6Q5U5SJ3FR7NDLYZDUWW2OEVYPYZMRNA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: VPRYCGGZTSNGK56O24LEYP4D2G5OFNEWGN6LMQZ5MRXU6IWC7MNQ) context deadline exceeded ======LOGS====== Dec 16 13:07:29.809079313 (261h38m29.680162358s) [upload=229405]: Starting upload Dec 16 13:07:29.809132684 (261h38m29.680215729s) [upload=229405]: Done starting upload Dec 16 13:07:29.809229719 (261h38m29.680312764s) [upload=229405]: Uploading segments... Dec 16 13:07:29.809330270 (261h38m29.680413315s) [upload=229405]: Got next segment. Inline: false Dec 16 13:07:29.809384634 (261h38m29.680467679s) [upload=229405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 13:07:29.809389824 (261h38m29.680472869s) [upload=229405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 13:07:30.059892657 (261h38m29.930975703s) [seg_pos={0 0}, upload=229405]: Begin upload segment... Dec 16 13:07:30.059941670 (261h38m29.931024705s) [seg_pos={0 0}, upload=229405]: Done begin upload segment. Dec 16 13:07:30.059972579 (261h38m29.931055614s) [noise=true , address=145.79.227.28:4727 , piece_id=2YVVE62BVFWW3Y7KIK6TTCGOAXJAP46ZHZP7WK7GVJYINVDGIXPA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.060303139 (261h38m29.931386174s) [noise=true , address=145.79.227.45:4229 , piece_id=4DKXTZ7CIR7U6OEPEFHOESXQDLXQULPOUMPWSXJAT67F4DGYD65Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.060779526 (261h38m29.931862561s) [noise=true , address=45.91.116.10:4082 , piece_id=54WRDBTLEJJO4ZKW2XQR5VUTU22IZ3QGQHPA3UXOGQNOAAHFTSGA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.061211679 (261h38m29.932294714s) [noise=true , address=77.90.31.11:4554 , piece_id=5TJQPKSFYLUVYTO3Y5QXTOIJ7IMW47DDRSX5PLPSDE6ZSEYHVWLA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.061662678 (261h38m29.932745723s) [noise=true , address=145.79.227.47:4266 , piece_id=QOPLAZEANWWZIX4IYYZ3C45OV7KTAZK5MXF732OVKTXMAXGSHT4A, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.062124056 (261h38m29.933207091s) [noise=true , address=145.79.227.39:4013 , piece_id=RA4C4QVO6SPSJCYQOFX6C7FOFM7U4MFLGYV2LIYSNOBVV5RFMVRQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.062633295 (261h38m29.93371634s) [noise=true , address=145.79.227.27:4691 , piece_id=UJNS3TFU57M36BSJAVY262VTNQ4OYOFRLYZ44FYOPGPFBXC5F6DA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.063080407 (261h38m29.934163452s) [noise=true , address=77.90.31.14:4663 , piece_id=AIM5PKBTA4YZI3THAPNQ47RWB3YROEZLQXMFYNE4TAGDWEYX5I6Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.064215909 (261h38m29.935298955s) [noise=true , address=145.79.227.41:4086 , piece_id=VLXONMHQF6NUO34NJMVIW23WZ2LLLPLELTPKX4D67N23M6NASB7Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.064624197 (261h38m29.935707242s) [noise=true , address=45.91.116.6:4976 , piece_id=AJCQN3JS34CN7NTABZFI6Q5U5SJ3FR7NDLYZDUWW2OEVYPYZMRNA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.065349709 (261h38m29.936432744s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ3X5T4LI4LEEXOOMBBQI2QIFHYL2W6PGBUKHOK6R7RPE643MQKQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.066606241 (261h38m29.937689276s) [noise=true , address=45.91.116.8:4046 , piece_id=VPRYCGGZTSNGK56O24LEYP4D2G5OFNEWGN6LMQZ5MRXU6IWC7MNQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.068358207 (261h38m29.939441242s) [noise=true , address=145.79.227.51:4411 , piece_id=RFJUQ7IHEZKKBZUMP5QJPVTZGL3XKZ6Z3Q7GH6FAYLA7T2GV4ZDQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.069461298 (261h38m29.940544343s) [noise=true , address=77.90.31.13:4629 , piece_id=JVQVLF6LJFPXZQETSJSISX7JVIQEP2MV523XX6IW6GV6J6YHH3LA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.071149843 (261h38m29.942232888s) [noise=true , address=145.79.227.48:4301 , piece_id=GJ6KJOLKZWRYR2WKYXLTM7X7EW4PD5TAZHHDBUDKORRQ3ZCKLDYA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.073241445 (261h38m29.94432448s) [noise=true , address=145.79.227.35:4482 , piece_id=IHGPOLNU6UIIE7JHUVQAGG7EXKVPL3H2XMVBJB62QNW2YXA3TI7Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:31.687109867 (261h38m31.558192912s) [upload=229405]: Next returned nil segment Dec 16 13:07:31.687122601 (261h38m31.558205636s) [upload=229405]: Waiting for error group managing segments... Dec 16 13:07:32.063153543 (261h38m31.934236588s) [noise=true , address=145.79.227.47:4266 , piece_id=QOPLAZEANWWZIX4IYYZ3C45OV7KTAZK5MXF732OVKTXMAXGSHT4A, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.064028158 (261h38m31.935111193s) [noise=true , address=145.79.227.41:4086 , piece_id=VLXONMHQF6NUO34NJMVIW23WZ2LLLPLELTPKX4D67N23M6NASB7Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.065193597 (261h38m31.936276642s) [noise=true , address=145.79.227.51:4411 , piece_id=RFJUQ7IHEZKKBZUMP5QJPVTZGL3XKZ6Z3Q7GH6FAYLA7T2GV4ZDQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.067197122 (261h38m31.938280167s) [noise=true , address=145.79.227.28:4727 , piece_id=2YVVE62BVFWW3Y7KIK6TTCGOAXJAP46ZHZP7WK7GVJYINVDGIXPA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.080675828 (261h38m31.951758863s) [noise=true , address=145.79.227.39:4013 , piece_id=RA4C4QVO6SPSJCYQOFX6C7FOFM7U4MFLGYV2LIYSNOBVV5RFMVRQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.084687868 (261h38m31.955770903s) [noise=true , address=145.79.227.48:4301 , piece_id=GJ6KJOLKZWRYR2WKYXLTM7X7EW4PD5TAZHHDBUDKORRQ3ZCKLDYA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.086130906 (261h38m31.957213951s) [noise=true , address=145.79.227.35:4482 , piece_id=IHGPOLNU6UIIE7JHUVQAGG7EXKVPL3H2XMVBJB62QNW2YXA3TI7Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.128559507 (261h38m31.999642552s) [noise=true , address=145.79.227.45:4229 , piece_id=4DKXTZ7CIR7U6OEPEFHOESXQDLXQULPOUMPWSXJAT67F4DGYD65Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.154672822 (261h38m32.025755857s) [noise=true , address=145.79.227.27:4691 , piece_id=UJNS3TFU57M36BSJAVY262VTNQ4OYOFRLYZ44FYOPGPFBXC5F6DA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.289498536 (261h38m32.160581581s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ3X5T4LI4LEEXOOMBBQI2QIFHYL2W6PGBUKHOK6R7RPE643MQKQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.294637029 (261h38m32.165720075s) [noise=true , address=45.91.116.6:4976 , piece_id=AJCQN3JS34CN7NTABZFI6Q5U5SJ3FR7NDLYZDUWW2OEVYPYZMRNA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.329777794 (261h38m32.200860829s) [noise=true , address=45.91.116.8:4046 , piece_id=VPRYCGGZTSNGK56O24LEYP4D2G5OFNEWGN6LMQZ5MRXU6IWC7MNQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.343192228 (261h38m32.214275263s) [noise=true , address=77.90.31.14:4663 , piece_id=AIM5PKBTA4YZI3THAPNQ47RWB3YROEZLQXMFYNE4TAGDWEYX5I6Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.343221523 (261h38m32.214304558s) [seg_pos={0 0}, upload=229405]: Segment reached optimal threshold of 13 pieces. Dec 16 13:07:32.343456340 (261h38m32.214539386s) [noise=true , address=77.90.31.13:4629 , piece_id=JVQVLF6LJFPXZQETSJSISX7JVIQEP2MV523XX6IW6GV6J6YHH3LA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 16 13:07:32.343467192 (261h38m32.214550227s) [noise=true , address=77.90.31.13:4629 , piece_id=JVQVLF6LJFPXZQETSJSISX7JVIQEP2MV523XX6IW6GV6J6YHH3LA, seg_pos={0 0}, upload=229405]: Long tail cancellation due to optimal threshold Dec 16 13:07:32.343856383 (261h38m32.214939418s) [noise=true , address=77.90.31.11:4554 , piece_id=5TJQPKSFYLUVYTO3Y5QXTOIJ7IMW47DDRSX5PLPSDE6ZSEYHVWLA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 16 13:07:32.343867413 (261h38m32.214950448s) [noise=true , address=77.90.31.11:4554 , piece_id=5TJQPKSFYLUVYTO3Y5QXTOIJ7IMW47DDRSX5PLPSDE6ZSEYHVWLA, seg_pos={0 0}, upload=229405]: Long tail cancellation due to optimal threshold Dec 16 13:07:32.344100427 (261h38m32.215183472s) [noise=true , address=45.91.116.10:4082 , piece_id=54WRDBTLEJJO4ZKW2XQR5VUTU22IZ3QGQHPA3UXOGQNOAAHFTSGA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 13:07:32.344106148 (261h38m32.215189183s) [noise=true , address=45.91.116.10:4082 , piece_id=54WRDBTLEJJO4ZKW2XQR5VUTU22IZ3QGQHPA3UXOGQNOAAHFTSGA, seg_pos={0 0}, upload=229405]: Long tail cancellation due to optimal threshold Dec 16 13:07:32.344120625 (261h38m32.21520367s) [seg_pos={0 0}, upload=229405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 13:07:32.344181852 (261h38m32.215264887s) [upload=229405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:07:32.344186010 (261h38m32.215269055s) [upload=229405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:07:32.636405379 (261h38m32.507488434s) [upload=229405]: Done uploading segments... Dec 16 13:07:32.636428022 (261h38m32.507511067s) [upload=229405]: Upload finished. err: <nil> Duration: 6.018556187s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [.X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.XX.X..X..] duration: 4.860450262s trace: https://jaeger.dev.storj.io/trace/4588d57ae9f8ccb5 FAIL (2025-12-16 13:08:43 UTC) orbiter warning: context deadline exceeded Duration: 4.860476612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X..X..........................X.....X.] duration: 5.946454436s trace: https://jaeger.dev.storj.io/trace/5305a9e5a4c67493 FAIL (2025-12-16 13:08:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: AJCQN3JS34CN7NTABZFI6Q5U5SJ3FR7NDLYZDUWW2OEVYPYZMRNA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: VPRYCGGZTSNGK56O24LEYP4D2G5OFNEWGN6LMQZ5MRXU6IWC7MNQ) context deadline exceeded ======LOGS====== Dec 16 13:07:29.809079313 (261h38m29.680162358s) [upload=229405]: Starting upload Dec 16 13:07:29.809132684 (261h38m29.680215729s) [upload=229405]: Done starting upload Dec 16 13:07:29.809229719 (261h38m29.680312764s) [upload=229405]: Uploading segments... Dec 16 13:07:29.809330270 (261h38m29.680413315s) [upload=229405]: Got next segment. Inline: false Dec 16 13:07:29.809384634 (261h38m29.680467679s) [upload=229405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 13:07:29.809389824 (261h38m29.680472869s) [upload=229405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 13:07:30.059892657 (261h38m29.930975703s) [seg_pos={0 0}, upload=229405]: Begin upload segment... Dec 16 13:07:30.059941670 (261h38m29.931024705s) [seg_pos={0 0}, upload=229405]: Done begin upload segment. Dec 16 13:07:30.059972579 (261h38m29.931055614s) [noise=true , address=145.79.227.28:4727 , piece_id=2YVVE62BVFWW3Y7KIK6TTCGOAXJAP46ZHZP7WK7GVJYINVDGIXPA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.060303139 (261h38m29.931386174s) [noise=true , address=145.79.227.45:4229 , piece_id=4DKXTZ7CIR7U6OEPEFHOESXQDLXQULPOUMPWSXJAT67F4DGYD65Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.060779526 (261h38m29.931862561s) [noise=true , address=45.91.116.10:4082 , piece_id=54WRDBTLEJJO4ZKW2XQR5VUTU22IZ3QGQHPA3UXOGQNOAAHFTSGA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.061211679 (261h38m29.932294714s) [noise=true , address=77.90.31.11:4554 , piece_id=5TJQPKSFYLUVYTO3Y5QXTOIJ7IMW47DDRSX5PLPSDE6ZSEYHVWLA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.061662678 (261h38m29.932745723s) [noise=true , address=145.79.227.47:4266 , piece_id=QOPLAZEANWWZIX4IYYZ3C45OV7KTAZK5MXF732OVKTXMAXGSHT4A, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.062124056 (261h38m29.933207091s) [noise=true , address=145.79.227.39:4013 , piece_id=RA4C4QVO6SPSJCYQOFX6C7FOFM7U4MFLGYV2LIYSNOBVV5RFMVRQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.062633295 (261h38m29.93371634s) [noise=true , address=145.79.227.27:4691 , piece_id=UJNS3TFU57M36BSJAVY262VTNQ4OYOFRLYZ44FYOPGPFBXC5F6DA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.063080407 (261h38m29.934163452s) [noise=true , address=77.90.31.14:4663 , piece_id=AIM5PKBTA4YZI3THAPNQ47RWB3YROEZLQXMFYNE4TAGDWEYX5I6Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.064215909 (261h38m29.935298955s) [noise=true , address=145.79.227.41:4086 , piece_id=VLXONMHQF6NUO34NJMVIW23WZ2LLLPLELTPKX4D67N23M6NASB7Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.064624197 (261h38m29.935707242s) [noise=true , address=45.91.116.6:4976 , piece_id=AJCQN3JS34CN7NTABZFI6Q5U5SJ3FR7NDLYZDUWW2OEVYPYZMRNA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.065349709 (261h38m29.936432744s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ3X5T4LI4LEEXOOMBBQI2QIFHYL2W6PGBUKHOK6R7RPE643MQKQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.066606241 (261h38m29.937689276s) [noise=true , address=45.91.116.8:4046 , piece_id=VPRYCGGZTSNGK56O24LEYP4D2G5OFNEWGN6LMQZ5MRXU6IWC7MNQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.068358207 (261h38m29.939441242s) [noise=true , address=145.79.227.51:4411 , piece_id=RFJUQ7IHEZKKBZUMP5QJPVTZGL3XKZ6Z3Q7GH6FAYLA7T2GV4ZDQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.069461298 (261h38m29.940544343s) [noise=true , address=77.90.31.13:4629 , piece_id=JVQVLF6LJFPXZQETSJSISX7JVIQEP2MV523XX6IW6GV6J6YHH3LA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.071149843 (261h38m29.942232888s) [noise=true , address=145.79.227.48:4301 , piece_id=GJ6KJOLKZWRYR2WKYXLTM7X7EW4PD5TAZHHDBUDKORRQ3ZCKLDYA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.073241445 (261h38m29.94432448s) [noise=true , address=145.79.227.35:4482 , piece_id=IHGPOLNU6UIIE7JHUVQAGG7EXKVPL3H2XMVBJB62QNW2YXA3TI7Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:31.687109867 (261h38m31.558192912s) [upload=229405]: Next returned nil segment Dec 16 13:07:31.687122601 (261h38m31.558205636s) [upload=229405]: Waiting for error group managing segments... Dec 16 13:07:32.063153543 (261h38m31.934236588s) [noise=true , address=145.79.227.47:4266 , piece_id=QOPLAZEANWWZIX4IYYZ3C45OV7KTAZK5MXF732OVKTXMAXGSHT4A, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.064028158 (261h38m31.935111193s) [noise=true , address=145.79.227.41:4086 , piece_id=VLXONMHQF6NUO34NJMVIW23WZ2LLLPLELTPKX4D67N23M6NASB7Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.065193597 (261h38m31.936276642s) [noise=true , address=145.79.227.51:4411 , piece_id=RFJUQ7IHEZKKBZUMP5QJPVTZGL3XKZ6Z3Q7GH6FAYLA7T2GV4ZDQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.067197122 (261h38m31.938280167s) [noise=true , address=145.79.227.28:4727 , piece_id=2YVVE62BVFWW3Y7KIK6TTCGOAXJAP46ZHZP7WK7GVJYINVDGIXPA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.080675828 (261h38m31.951758863s) [noise=true , address=145.79.227.39:4013 , piece_id=RA4C4QVO6SPSJCYQOFX6C7FOFM7U4MFLGYV2LIYSNOBVV5RFMVRQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.084687868 (261h38m31.955770903s) [noise=true , address=145.79.227.48:4301 , piece_id=GJ6KJOLKZWRYR2WKYXLTM7X7EW4PD5TAZHHDBUDKORRQ3ZCKLDYA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.086130906 (261h38m31.957213951s) [noise=true , address=145.79.227.35:4482 , piece_id=IHGPOLNU6UIIE7JHUVQAGG7EXKVPL3H2XMVBJB62QNW2YXA3TI7Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.128559507 (261h38m31.999642552s) [noise=true , address=145.79.227.45:4229 , piece_id=4DKXTZ7CIR7U6OEPEFHOESXQDLXQULPOUMPWSXJAT67F4DGYD65Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.154672822 (261h38m32.025755857s) [noise=true , address=145.79.227.27:4691 , piece_id=UJNS3TFU57M36BSJAVY262VTNQ4OYOFRLYZ44FYOPGPFBXC5F6DA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.289498536 (261h38m32.160581581s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ3X5T4LI4LEEXOOMBBQI2QIFHYL2W6PGBUKHOK6R7RPE643MQKQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.294637029 (261h38m32.165720075s) [noise=true , address=45.91.116.6:4976 , piece_id=AJCQN3JS34CN7NTABZFI6Q5U5SJ3FR7NDLYZDUWW2OEVYPYZMRNA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.329777794 (261h38m32.200860829s) [noise=true , address=45.91.116.8:4046 , piece_id=VPRYCGGZTSNGK56O24LEYP4D2G5OFNEWGN6LMQZ5MRXU6IWC7MNQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.343192228 (261h38m32.214275263s) [noise=true , address=77.90.31.14:4663 , piece_id=AIM5PKBTA4YZI3THAPNQ47RWB3YROEZLQXMFYNE4TAGDWEYX5I6Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.343221523 (261h38m32.214304558s) [seg_pos={0 0}, upload=229405]: Segment reached optimal threshold of 13 pieces. Dec 16 13:07:32.343456340 (261h38m32.214539386s) [noise=true , address=77.90.31.13:4629 , piece_id=JVQVLF6LJFPXZQETSJSISX7JVIQEP2MV523XX6IW6GV6J6YHH3LA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 16 13:07:32.343467192 (261h38m32.214550227s) [noise=true , address=77.90.31.13:4629 , piece_id=JVQVLF6LJFPXZQETSJSISX7JVIQEP2MV523XX6IW6GV6J6YHH3LA, seg_pos={0 0}, upload=229405]: Long tail cancellation due to optimal threshold Dec 16 13:07:32.343856383 (261h38m32.214939418s) [noise=true , address=77.90.31.11:4554 , piece_id=5TJQPKSFYLUVYTO3Y5QXTOIJ7IMW47DDRSX5PLPSDE6ZSEYHVWLA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 16 13:07:32.343867413 (261h38m32.214950448s) [noise=true , address=77.90.31.11:4554 , piece_id=5TJQPKSFYLUVYTO3Y5QXTOIJ7IMW47DDRSX5PLPSDE6ZSEYHVWLA, seg_pos={0 0}, upload=229405]: Long tail cancellation due to optimal threshold Dec 16 13:07:32.344100427 (261h38m32.215183472s) [noise=true , address=45.91.116.10:4082 , piece_id=54WRDBTLEJJO4ZKW2XQR5VUTU22IZ3QGQHPA3UXOGQNOAAHFTSGA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 13:07:32.344106148 (261h38m32.215189183s) [noise=true , address=45.91.116.10:4082 , piece_id=54WRDBTLEJJO4ZKW2XQR5VUTU22IZ3QGQHPA3UXOGQNOAAHFTSGA, seg_pos={0 0}, upload=229405]: Long tail cancellation due to optimal threshold Dec 16 13:07:32.344120625 (261h38m32.21520367s) [seg_pos={0 0}, upload=229405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 13:07:32.344181852 (261h38m32.215264887s) [upload=229405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:07:32.344186010 (261h38m32.215269055s) [upload=229405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:07:32.636405379 (261h38m32.507488434s) [upload=229405]: Done uploading segments... Dec 16 13:07:32.636428022 (261h38m32.507511067s) [upload=229405]: Upload finished. err: <nil> Duration: 5.946489642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X..X..........................X.....X] duration: 30.003617696s trace: https://jaeger.dev.storj.io/trace/5305a9e5a4c67493 FAIL (2025-12-16 13:07:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: AJCQN3JS34CN7NTABZFI6Q5U5SJ3FR7NDLYZDUWW2OEVYPYZMRNA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: VPRYCGGZTSNGK56O24LEYP4D2G5OFNEWGN6LMQZ5MRXU6IWC7MNQ) context deadline exceeded ======LOGS====== Dec 16 13:07:29.809079313 (261h38m29.680162358s) [upload=229405]: Starting upload Dec 16 13:07:29.809132684 (261h38m29.680215729s) [upload=229405]: Done starting upload Dec 16 13:07:29.809229719 (261h38m29.680312764s) [upload=229405]: Uploading segments... Dec 16 13:07:29.809330270 (261h38m29.680413315s) [upload=229405]: Got next segment. Inline: false Dec 16 13:07:29.809384634 (261h38m29.680467679s) [upload=229405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 13:07:29.809389824 (261h38m29.680472869s) [upload=229405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 13:07:30.059892657 (261h38m29.930975703s) [seg_pos={0 0}, upload=229405]: Begin upload segment... Dec 16 13:07:30.059941670 (261h38m29.931024705s) [seg_pos={0 0}, upload=229405]: Done begin upload segment. Dec 16 13:07:30.059972579 (261h38m29.931055614s) [noise=true , address=145.79.227.28:4727 , piece_id=2YVVE62BVFWW3Y7KIK6TTCGOAXJAP46ZHZP7WK7GVJYINVDGIXPA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.060303139 (261h38m29.931386174s) [noise=true , address=145.79.227.45:4229 , piece_id=4DKXTZ7CIR7U6OEPEFHOESXQDLXQULPOUMPWSXJAT67F4DGYD65Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.060779526 (261h38m29.931862561s) [noise=true , address=45.91.116.10:4082 , piece_id=54WRDBTLEJJO4ZKW2XQR5VUTU22IZ3QGQHPA3UXOGQNOAAHFTSGA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.061211679 (261h38m29.932294714s) [noise=true , address=77.90.31.11:4554 , piece_id=5TJQPKSFYLUVYTO3Y5QXTOIJ7IMW47DDRSX5PLPSDE6ZSEYHVWLA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.061662678 (261h38m29.932745723s) [noise=true , address=145.79.227.47:4266 , piece_id=QOPLAZEANWWZIX4IYYZ3C45OV7KTAZK5MXF732OVKTXMAXGSHT4A, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.062124056 (261h38m29.933207091s) [noise=true , address=145.79.227.39:4013 , piece_id=RA4C4QVO6SPSJCYQOFX6C7FOFM7U4MFLGYV2LIYSNOBVV5RFMVRQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.062633295 (261h38m29.93371634s) [noise=true , address=145.79.227.27:4691 , piece_id=UJNS3TFU57M36BSJAVY262VTNQ4OYOFRLYZ44FYOPGPFBXC5F6DA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.063080407 (261h38m29.934163452s) [noise=true , address=77.90.31.14:4663 , piece_id=AIM5PKBTA4YZI3THAPNQ47RWB3YROEZLQXMFYNE4TAGDWEYX5I6Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.064215909 (261h38m29.935298955s) [noise=true , address=145.79.227.41:4086 , piece_id=VLXONMHQF6NUO34NJMVIW23WZ2LLLPLELTPKX4D67N23M6NASB7Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.064624197 (261h38m29.935707242s) [noise=true , address=45.91.116.6:4976 , piece_id=AJCQN3JS34CN7NTABZFI6Q5U5SJ3FR7NDLYZDUWW2OEVYPYZMRNA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.065349709 (261h38m29.936432744s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ3X5T4LI4LEEXOOMBBQI2QIFHYL2W6PGBUKHOK6R7RPE643MQKQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.066606241 (261h38m29.937689276s) [noise=true , address=45.91.116.8:4046 , piece_id=VPRYCGGZTSNGK56O24LEYP4D2G5OFNEWGN6LMQZ5MRXU6IWC7MNQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.068358207 (261h38m29.939441242s) [noise=true , address=145.79.227.51:4411 , piece_id=RFJUQ7IHEZKKBZUMP5QJPVTZGL3XKZ6Z3Q7GH6FAYLA7T2GV4ZDQ, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.069461298 (261h38m29.940544343s) [noise=true , address=77.90.31.13:4629 , piece_id=JVQVLF6LJFPXZQETSJSISX7JVIQEP2MV523XX6IW6GV6J6YHH3LA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.071149843 (261h38m29.942232888s) [noise=true , address=145.79.227.48:4301 , piece_id=GJ6KJOLKZWRYR2WKYXLTM7X7EW4PD5TAZHHDBUDKORRQ3ZCKLDYA, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:30.073241445 (261h38m29.94432448s) [noise=true , address=145.79.227.35:4482 , piece_id=IHGPOLNU6UIIE7JHUVQAGG7EXKVPL3H2XMVBJB62QNW2YXA3TI7Q, seg_pos={0 0}, upload=229405]: Uploading piece... Dec 16 13:07:31.687109867 (261h38m31.558192912s) [upload=229405]: Next returned nil segment Dec 16 13:07:31.687122601 (261h38m31.558205636s) [upload=229405]: Waiting for error group managing segments... Dec 16 13:07:32.063153543 (261h38m31.934236588s) [noise=true , address=145.79.227.47:4266 , piece_id=QOPLAZEANWWZIX4IYYZ3C45OV7KTAZK5MXF732OVKTXMAXGSHT4A, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.064028158 (261h38m31.935111193s) [noise=true , address=145.79.227.41:4086 , piece_id=VLXONMHQF6NUO34NJMVIW23WZ2LLLPLELTPKX4D67N23M6NASB7Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.065193597 (261h38m31.936276642s) [noise=true , address=145.79.227.51:4411 , piece_id=RFJUQ7IHEZKKBZUMP5QJPVTZGL3XKZ6Z3Q7GH6FAYLA7T2GV4ZDQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.067197122 (261h38m31.938280167s) [noise=true , address=145.79.227.28:4727 , piece_id=2YVVE62BVFWW3Y7KIK6TTCGOAXJAP46ZHZP7WK7GVJYINVDGIXPA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.080675828 (261h38m31.951758863s) [noise=true , address=145.79.227.39:4013 , piece_id=RA4C4QVO6SPSJCYQOFX6C7FOFM7U4MFLGYV2LIYSNOBVV5RFMVRQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.084687868 (261h38m31.955770903s) [noise=true , address=145.79.227.48:4301 , piece_id=GJ6KJOLKZWRYR2WKYXLTM7X7EW4PD5TAZHHDBUDKORRQ3ZCKLDYA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.086130906 (261h38m31.957213951s) [noise=true , address=145.79.227.35:4482 , piece_id=IHGPOLNU6UIIE7JHUVQAGG7EXKVPL3H2XMVBJB62QNW2YXA3TI7Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.128559507 (261h38m31.999642552s) [noise=true , address=145.79.227.45:4229 , piece_id=4DKXTZ7CIR7U6OEPEFHOESXQDLXQULPOUMPWSXJAT67F4DGYD65Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.154672822 (261h38m32.025755857s) [noise=true , address=145.79.227.27:4691 , piece_id=UJNS3TFU57M36BSJAVY262VTNQ4OYOFRLYZ44FYOPGPFBXC5F6DA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.289498536 (261h38m32.160581581s) [noise=true , address=77.90.31.17:4773 , piece_id=CJ3X5T4LI4LEEXOOMBBQI2QIFHYL2W6PGBUKHOK6R7RPE643MQKQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.294637029 (261h38m32.165720075s) [noise=true , address=45.91.116.6:4976 , piece_id=AJCQN3JS34CN7NTABZFI6Q5U5SJ3FR7NDLYZDUWW2OEVYPYZMRNA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.329777794 (261h38m32.200860829s) [noise=true , address=45.91.116.8:4046 , piece_id=VPRYCGGZTSNGK56O24LEYP4D2G5OFNEWGN6LMQZ5MRXU6IWC7MNQ, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.343192228 (261h38m32.214275263s) [noise=true , address=77.90.31.14:4663 , piece_id=AIM5PKBTA4YZI3THAPNQ47RWB3YROEZLQXMFYNE4TAGDWEYX5I6Q, seg_pos={0 0}, upload=229405]: Done uploading piece. err: <nil> Dec 16 13:07:32.343221523 (261h38m32.214304558s) [seg_pos={0 0}, upload=229405]: Segment reached optimal threshold of 13 pieces. Dec 16 13:07:32.343456340 (261h38m32.214539386s) [noise=true , address=77.90.31.13:4629 , piece_id=JVQVLF6LJFPXZQETSJSISX7JVIQEP2MV523XX6IW6GV6J6YHH3LA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 16 13:07:32.343467192 (261h38m32.214550227s) [noise=true , address=77.90.31.13:4629 , piece_id=JVQVLF6LJFPXZQETSJSISX7JVIQEP2MV523XX6IW6GV6J6YHH3LA, seg_pos={0 0}, upload=229405]: Long tail cancellation due to optimal threshold Dec 16 13:07:32.343856383 (261h38m32.214939418s) [noise=true , address=77.90.31.11:4554 , piece_id=5TJQPKSFYLUVYTO3Y5QXTOIJ7IMW47DDRSX5PLPSDE6ZSEYHVWLA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 16 13:07:32.343867413 (261h38m32.214950448s) [noise=true , address=77.90.31.11:4554 , piece_id=5TJQPKSFYLUVYTO3Y5QXTOIJ7IMW47DDRSX5PLPSDE6ZSEYHVWLA, seg_pos={0 0}, upload=229405]: Long tail cancellation due to optimal threshold Dec 16 13:07:32.344100427 (261h38m32.215183472s) [noise=true , address=45.91.116.10:4082 , piece_id=54WRDBTLEJJO4ZKW2XQR5VUTU22IZ3QGQHPA3UXOGQNOAAHFTSGA, seg_pos={0 0}, upload=229405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 16 13:07:32.344106148 (261h38m32.215189183s) [noise=true , address=45.91.116.10:4082 , piece_id=54WRDBTLEJJO4ZKW2XQR5VUTU22IZ3QGQHPA3UXOGQNOAAHFTSGA, seg_pos={0 0}, upload=229405]: Long tail cancellation due to optimal threshold Dec 16 13:07:32.344120625 (261h38m32.21520367s) [seg_pos={0 0}, upload=229405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 13:07:32.344181852 (261h38m32.215264887s) [upload=229405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:07:32.344186010 (261h38m32.215269055s) [upload=229405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:07:32.636405379 (261h38m32.507488434s) [upload=229405]: Done uploading segments... Dec 16 13:07:32.636428022 (261h38m32.507511067s) [upload=229405]: Upload finished. err: <nil> Duration: 30.003648064s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [..X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.XX.X..X.] duration: 3.751059105s trace: https://jaeger.dev.storj.io/trace/4588d57ae9f8ccb5 FAIL (2025-12-16 13:07:29 UTC) orbiter warning: context deadline exceeded Duration: 3.751087479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X..X..........................X.....] duration: 5.907875241s trace: https://jaeger.dev.storj.io/trace/1269ca34c62795ac FAIL (2025-12-16 13:07:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A) context deadline exceeded ======LOGS====== Dec 16 13:03:27.325733314 (261h34m27.196816359s) [upload=229345]: Starting upload Dec 16 13:03:27.325826892 (261h34m27.196909927s) [upload=229345]: Done starting upload Dec 16 13:03:27.325896174 (261h34m27.196979209s) [upload=229345]: Uploading segments... Dec 16 13:03:27.326032604 (261h34m27.197115639s) [upload=229345]: Got next segment. Inline: false Dec 16 13:03:27.326061419 (261h34m27.197144454s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 13:03:27.326065917 (261h34m27.197148962s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 13:03:27.579709261 (261h34m27.450792306s) [seg_pos={0 0}, upload=229345]: Begin upload segment... Dec 16 13:03:27.579786128 (261h34m27.450869173s) [seg_pos={0 0}, upload=229345]: Done begin upload segment. Dec 16 13:03:27.579877792 (261h34m27.450960827s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.579884544 (261h34m27.450967589s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.580642618 (261h34m27.451725653s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581050575 (261h34m27.45213361s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581345797 (261h34m27.452428832s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581658763 (261h34m27.452741798s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581985996 (261h34m27.453069031s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582269486 (261h34m27.453352521s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582756183 (261h34m27.453839218s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.583917484 (261h34m27.455000519s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.584013868 (261h34m27.455096913s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585074227 (261h34m27.456157262s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585212761 (261h34m27.456295796s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586387006 (261h34m27.457470091s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586452481 (261h34m27.457535516s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.587432337 (261h34m27.458515372s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:29.291347766 (261h34m29.162430811s) [upload=229345]: Next returned nil segment Dec 16 13:03:29.291360249 (261h34m29.162443294s) [upload=229345]: Waiting for error group managing segments... Dec 16 13:03:29.574649955 (261h34m29.445733s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.577336752 (261h34m29.448419787s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.578807943 (261h34m29.449890988s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.595546666 (261h34m29.466629711s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.596591276 (261h34m29.467674331s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599217586 (261h34m29.470300621s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599865620 (261h34m29.470948665s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.601142182 (261h34m29.472225227s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.719001928 (261h34m29.590084963s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.721030170 (261h34m29.592113205s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.737244916 (261h34m29.608327961s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.796723894 (261h34m29.667806929s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898268494 (261h34m29.769351539s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898384775 (261h34m29.76946781s) [seg_pos={0 0}, upload=229345]: Segment reached optimal threshold of 13 pieces. Dec 16 13:03:29.898653948 (261h34m29.769736993s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 16 13:03:29.898663225 (261h34m29.76974626s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898837377 (261h34m29.769920422s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:60838->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 13:03:29.898879627 (261h34m29.769962672s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898997452 (261h34m29.770080497s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:50604->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 16 13:03:29.899006409 (261h34m29.770089454s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.899028631 (261h34m29.770111676s) [seg_pos={0 0}, upload=229345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 13:03:29.899109956 (261h34m29.770192991s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:03:29.899117520 (261h34m29.770200565s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:03:30.213746944 (261h34m30.084829979s) [upload=229345]: Done uploading segments... Dec 16 13:03:30.213767122 (261h34m30.084850157s) [upload=229345]: Upload finished. err: <nil> Duration: 5.907901691s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [...X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.XX.X..X] duration: 5.000995692s trace: https://jaeger.dev.storj.io/trace/4588d57ae9f8ccb5 FAIL (2025-12-16 13:07:14 UTC) orbiter warning: context deadline exceeded Duration: 5.001037271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..X..........................X....] duration: 6.080057597s trace: https://jaeger.dev.storj.io/trace/1269ca34c62795ac FAIL (2025-12-16 13:06:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A) context deadline exceeded ======LOGS====== Dec 16 13:03:27.325733314 (261h34m27.196816359s) [upload=229345]: Starting upload Dec 16 13:03:27.325826892 (261h34m27.196909927s) [upload=229345]: Done starting upload Dec 16 13:03:27.325896174 (261h34m27.196979209s) [upload=229345]: Uploading segments... Dec 16 13:03:27.326032604 (261h34m27.197115639s) [upload=229345]: Got next segment. Inline: false Dec 16 13:03:27.326061419 (261h34m27.197144454s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 13:03:27.326065917 (261h34m27.197148962s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 13:03:27.579709261 (261h34m27.450792306s) [seg_pos={0 0}, upload=229345]: Begin upload segment... Dec 16 13:03:27.579786128 (261h34m27.450869173s) [seg_pos={0 0}, upload=229345]: Done begin upload segment. Dec 16 13:03:27.579877792 (261h34m27.450960827s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.579884544 (261h34m27.450967589s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.580642618 (261h34m27.451725653s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581050575 (261h34m27.45213361s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581345797 (261h34m27.452428832s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581658763 (261h34m27.452741798s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581985996 (261h34m27.453069031s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582269486 (261h34m27.453352521s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582756183 (261h34m27.453839218s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.583917484 (261h34m27.455000519s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.584013868 (261h34m27.455096913s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585074227 (261h34m27.456157262s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585212761 (261h34m27.456295796s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586387006 (261h34m27.457470091s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586452481 (261h34m27.457535516s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.587432337 (261h34m27.458515372s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:29.291347766 (261h34m29.162430811s) [upload=229345]: Next returned nil segment Dec 16 13:03:29.291360249 (261h34m29.162443294s) [upload=229345]: Waiting for error group managing segments... Dec 16 13:03:29.574649955 (261h34m29.445733s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.577336752 (261h34m29.448419787s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.578807943 (261h34m29.449890988s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.595546666 (261h34m29.466629711s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.596591276 (261h34m29.467674331s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599217586 (261h34m29.470300621s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599865620 (261h34m29.470948665s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.601142182 (261h34m29.472225227s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.719001928 (261h34m29.590084963s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.721030170 (261h34m29.592113205s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.737244916 (261h34m29.608327961s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.796723894 (261h34m29.667806929s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898268494 (261h34m29.769351539s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898384775 (261h34m29.76946781s) [seg_pos={0 0}, upload=229345]: Segment reached optimal threshold of 13 pieces. Dec 16 13:03:29.898653948 (261h34m29.769736993s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 16 13:03:29.898663225 (261h34m29.76974626s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898837377 (261h34m29.769920422s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:60838->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 13:03:29.898879627 (261h34m29.769962672s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898997452 (261h34m29.770080497s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:50604->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 16 13:03:29.899006409 (261h34m29.770089454s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.899028631 (261h34m29.770111676s) [seg_pos={0 0}, upload=229345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 13:03:29.899109956 (261h34m29.770192991s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:03:29.899117520 (261h34m29.770200565s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:03:30.213746944 (261h34m30.084829979s) [upload=229345]: Done uploading segments... Dec 16 13:03:30.213767122 (261h34m30.084850157s) [upload=229345]: Upload finished. err: <nil> Duration: 6.080085249s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [....X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.XX.X..] duration: 3.748239407s trace: https://jaeger.dev.storj.io/trace/7db3bb4ee84af73 FAIL (2025-12-16 13:06:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.748263203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X..X..........................X...] duration: 5.68366912s trace: https://jaeger.dev.storj.io/trace/1269ca34c62795ac FAIL (2025-12-16 13:05:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A) context deadline exceeded ======LOGS====== Dec 16 13:03:27.325733314 (261h34m27.196816359s) [upload=229345]: Starting upload Dec 16 13:03:27.325826892 (261h34m27.196909927s) [upload=229345]: Done starting upload Dec 16 13:03:27.325896174 (261h34m27.196979209s) [upload=229345]: Uploading segments... Dec 16 13:03:27.326032604 (261h34m27.197115639s) [upload=229345]: Got next segment. Inline: false Dec 16 13:03:27.326061419 (261h34m27.197144454s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 13:03:27.326065917 (261h34m27.197148962s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 13:03:27.579709261 (261h34m27.450792306s) [seg_pos={0 0}, upload=229345]: Begin upload segment... Dec 16 13:03:27.579786128 (261h34m27.450869173s) [seg_pos={0 0}, upload=229345]: Done begin upload segment. Dec 16 13:03:27.579877792 (261h34m27.450960827s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.579884544 (261h34m27.450967589s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.580642618 (261h34m27.451725653s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581050575 (261h34m27.45213361s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581345797 (261h34m27.452428832s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581658763 (261h34m27.452741798s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581985996 (261h34m27.453069031s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582269486 (261h34m27.453352521s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582756183 (261h34m27.453839218s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.583917484 (261h34m27.455000519s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.584013868 (261h34m27.455096913s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585074227 (261h34m27.456157262s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585212761 (261h34m27.456295796s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586387006 (261h34m27.457470091s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586452481 (261h34m27.457535516s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.587432337 (261h34m27.458515372s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:29.291347766 (261h34m29.162430811s) [upload=229345]: Next returned nil segment Dec 16 13:03:29.291360249 (261h34m29.162443294s) [upload=229345]: Waiting for error group managing segments... Dec 16 13:03:29.574649955 (261h34m29.445733s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.577336752 (261h34m29.448419787s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.578807943 (261h34m29.449890988s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.595546666 (261h34m29.466629711s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.596591276 (261h34m29.467674331s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599217586 (261h34m29.470300621s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599865620 (261h34m29.470948665s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.601142182 (261h34m29.472225227s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.719001928 (261h34m29.590084963s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.721030170 (261h34m29.592113205s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.737244916 (261h34m29.608327961s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.796723894 (261h34m29.667806929s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898268494 (261h34m29.769351539s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898384775 (261h34m29.76946781s) [seg_pos={0 0}, upload=229345]: Segment reached optimal threshold of 13 pieces. Dec 16 13:03:29.898653948 (261h34m29.769736993s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 16 13:03:29.898663225 (261h34m29.76974626s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898837377 (261h34m29.769920422s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:60838->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 13:03:29.898879627 (261h34m29.769962672s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898997452 (261h34m29.770080497s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:50604->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 16 13:03:29.899006409 (261h34m29.770089454s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.899028631 (261h34m29.770111676s) [seg_pos={0 0}, upload=229345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 13:03:29.899109956 (261h34m29.770192991s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:03:29.899117520 (261h34m29.770200565s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:03:30.213746944 (261h34m30.084829979s) [upload=229345]: Done uploading segments... Dec 16 13:03:30.213767122 (261h34m30.084850157s) [upload=229345]: Upload finished. err: <nil> Duration: 5.683692846s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.....X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.XX.X.] duration: 4.219137306s trace: https://jaeger.dev.storj.io/trace/7db3bb4ee84af73 FAIL (2025-12-16 13:05:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.219171571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X..X..........................X..] duration: 6.085609218s trace: https://jaeger.dev.storj.io/trace/1269ca34c62795ac FAIL (2025-12-16 13:04:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A) context deadline exceeded ======LOGS====== Dec 16 13:03:27.325733314 (261h34m27.196816359s) [upload=229345]: Starting upload Dec 16 13:03:27.325826892 (261h34m27.196909927s) [upload=229345]: Done starting upload Dec 16 13:03:27.325896174 (261h34m27.196979209s) [upload=229345]: Uploading segments... Dec 16 13:03:27.326032604 (261h34m27.197115639s) [upload=229345]: Got next segment. Inline: false Dec 16 13:03:27.326061419 (261h34m27.197144454s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 13:03:27.326065917 (261h34m27.197148962s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 13:03:27.579709261 (261h34m27.450792306s) [seg_pos={0 0}, upload=229345]: Begin upload segment... Dec 16 13:03:27.579786128 (261h34m27.450869173s) [seg_pos={0 0}, upload=229345]: Done begin upload segment. Dec 16 13:03:27.579877792 (261h34m27.450960827s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.579884544 (261h34m27.450967589s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.580642618 (261h34m27.451725653s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581050575 (261h34m27.45213361s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581345797 (261h34m27.452428832s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581658763 (261h34m27.452741798s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581985996 (261h34m27.453069031s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582269486 (261h34m27.453352521s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582756183 (261h34m27.453839218s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.583917484 (261h34m27.455000519s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.584013868 (261h34m27.455096913s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585074227 (261h34m27.456157262s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585212761 (261h34m27.456295796s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586387006 (261h34m27.457470091s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586452481 (261h34m27.457535516s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.587432337 (261h34m27.458515372s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:29.291347766 (261h34m29.162430811s) [upload=229345]: Next returned nil segment Dec 16 13:03:29.291360249 (261h34m29.162443294s) [upload=229345]: Waiting for error group managing segments... Dec 16 13:03:29.574649955 (261h34m29.445733s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.577336752 (261h34m29.448419787s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.578807943 (261h34m29.449890988s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.595546666 (261h34m29.466629711s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.596591276 (261h34m29.467674331s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599217586 (261h34m29.470300621s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599865620 (261h34m29.470948665s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.601142182 (261h34m29.472225227s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.719001928 (261h34m29.590084963s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.721030170 (261h34m29.592113205s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.737244916 (261h34m29.608327961s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.796723894 (261h34m29.667806929s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898268494 (261h34m29.769351539s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898384775 (261h34m29.76946781s) [seg_pos={0 0}, upload=229345]: Segment reached optimal threshold of 13 pieces. Dec 16 13:03:29.898653948 (261h34m29.769736993s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 16 13:03:29.898663225 (261h34m29.76974626s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898837377 (261h34m29.769920422s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:60838->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 13:03:29.898879627 (261h34m29.769962672s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898997452 (261h34m29.770080497s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:50604->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 16 13:03:29.899006409 (261h34m29.770089454s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.899028631 (261h34m29.770111676s) [seg_pos={0 0}, upload=229345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 13:03:29.899109956 (261h34m29.770192991s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:03:29.899117520 (261h34m29.770200565s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:03:30.213746944 (261h34m30.084829979s) [upload=229345]: Done uploading segments... Dec 16 13:03:30.213767122 (261h34m30.084850157s) [upload=229345]: Upload finished. err: <nil> Duration: 6.085647512s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.XX.X] duration: 5.002168889s trace: https://jaeger.dev.storj.io/trace/7db3bb4ee84af73 FAIL (2025-12-16 13:04:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002194748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X..X..........................X.] duration: 5.984545625s trace: https://jaeger.dev.storj.io/trace/1269ca34c62795ac FAIL (2025-12-16 13:04:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A) context deadline exceeded ======LOGS====== Dec 16 13:03:27.325733314 (261h34m27.196816359s) [upload=229345]: Starting upload Dec 16 13:03:27.325826892 (261h34m27.196909927s) [upload=229345]: Done starting upload Dec 16 13:03:27.325896174 (261h34m27.196979209s) [upload=229345]: Uploading segments... Dec 16 13:03:27.326032604 (261h34m27.197115639s) [upload=229345]: Got next segment. Inline: false Dec 16 13:03:27.326061419 (261h34m27.197144454s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 13:03:27.326065917 (261h34m27.197148962s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 13:03:27.579709261 (261h34m27.450792306s) [seg_pos={0 0}, upload=229345]: Begin upload segment... Dec 16 13:03:27.579786128 (261h34m27.450869173s) [seg_pos={0 0}, upload=229345]: Done begin upload segment. Dec 16 13:03:27.579877792 (261h34m27.450960827s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.579884544 (261h34m27.450967589s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.580642618 (261h34m27.451725653s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581050575 (261h34m27.45213361s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581345797 (261h34m27.452428832s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581658763 (261h34m27.452741798s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581985996 (261h34m27.453069031s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582269486 (261h34m27.453352521s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582756183 (261h34m27.453839218s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.583917484 (261h34m27.455000519s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.584013868 (261h34m27.455096913s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585074227 (261h34m27.456157262s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585212761 (261h34m27.456295796s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586387006 (261h34m27.457470091s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586452481 (261h34m27.457535516s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.587432337 (261h34m27.458515372s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:29.291347766 (261h34m29.162430811s) [upload=229345]: Next returned nil segment Dec 16 13:03:29.291360249 (261h34m29.162443294s) [upload=229345]: Waiting for error group managing segments... Dec 16 13:03:29.574649955 (261h34m29.445733s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.577336752 (261h34m29.448419787s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.578807943 (261h34m29.449890988s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.595546666 (261h34m29.466629711s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.596591276 (261h34m29.467674331s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599217586 (261h34m29.470300621s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599865620 (261h34m29.470948665s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.601142182 (261h34m29.472225227s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.719001928 (261h34m29.590084963s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.721030170 (261h34m29.592113205s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.737244916 (261h34m29.608327961s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.796723894 (261h34m29.667806929s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898268494 (261h34m29.769351539s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898384775 (261h34m29.76946781s) [seg_pos={0 0}, upload=229345]: Segment reached optimal threshold of 13 pieces. Dec 16 13:03:29.898653948 (261h34m29.769736993s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 16 13:03:29.898663225 (261h34m29.76974626s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898837377 (261h34m29.769920422s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:60838->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 13:03:29.898879627 (261h34m29.769962672s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898997452 (261h34m29.770080497s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:50604->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 16 13:03:29.899006409 (261h34m29.770089454s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.899028631 (261h34m29.770111676s) [seg_pos={0 0}, upload=229345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 13:03:29.899109956 (261h34m29.770192991s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:03:29.899117520 (261h34m29.770200565s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:03:30.213746944 (261h34m30.084829979s) [upload=229345]: Done uploading segments... Dec 16 13:03:30.213767122 (261h34m30.084850157s) [upload=229345]: Upload finished. err: <nil> Duration: 5.984576403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X............................X..X..........................X] duration: 30.003488801s trace: https://jaeger.dev.storj.io/trace/1269ca34c62795ac FAIL (2025-12-16 13:03:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A) context deadline exceeded ======LOGS====== Dec 16 13:03:27.325733314 (261h34m27.196816359s) [upload=229345]: Starting upload Dec 16 13:03:27.325826892 (261h34m27.196909927s) [upload=229345]: Done starting upload Dec 16 13:03:27.325896174 (261h34m27.196979209s) [upload=229345]: Uploading segments... Dec 16 13:03:27.326032604 (261h34m27.197115639s) [upload=229345]: Got next segment. Inline: false Dec 16 13:03:27.326061419 (261h34m27.197144454s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 13:03:27.326065917 (261h34m27.197148962s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 13:03:27.579709261 (261h34m27.450792306s) [seg_pos={0 0}, upload=229345]: Begin upload segment... Dec 16 13:03:27.579786128 (261h34m27.450869173s) [seg_pos={0 0}, upload=229345]: Done begin upload segment. Dec 16 13:03:27.579877792 (261h34m27.450960827s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.579884544 (261h34m27.450967589s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.580642618 (261h34m27.451725653s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581050575 (261h34m27.45213361s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581345797 (261h34m27.452428832s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581658763 (261h34m27.452741798s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.581985996 (261h34m27.453069031s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582269486 (261h34m27.453352521s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.582756183 (261h34m27.453839218s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.583917484 (261h34m27.455000519s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.584013868 (261h34m27.455096913s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585074227 (261h34m27.456157262s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.585212761 (261h34m27.456295796s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586387006 (261h34m27.457470091s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.586452481 (261h34m27.457535516s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:27.587432337 (261h34m27.458515372s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Uploading piece... Dec 16 13:03:29.291347766 (261h34m29.162430811s) [upload=229345]: Next returned nil segment Dec 16 13:03:29.291360249 (261h34m29.162443294s) [upload=229345]: Waiting for error group managing segments... Dec 16 13:03:29.574649955 (261h34m29.445733s) [noise=true , address=145.79.227.49:4338 , piece_id=BN5OWWQEOID2WVZTB2ZC4V6RXEF6XTJUIOQRPLUWPVUOUHEAXMOA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.577336752 (261h34m29.448419787s) [noise=true , address=145.79.227.4:4185 , piece_id=DFZIAIFA34V5BVFBWB3TQDZU5TQE4MCHOZ2B3TGVEI37SPPI65GA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.578807943 (261h34m29.449890988s) [noise=true , address=145.79.227.26:4653 , piece_id=6F4BIIDI3B5YIOILJ4J6TNXQDQTSB76UWCPMB2Q5LGVXR4I2NQNQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.595546666 (261h34m29.466629711s) [noise=true , address=145.79.227.36:4517 , piece_id=4DD6MPOT6RX3CPURBDOXDRVIEJEZODAQTAAIL7G6B3O2WB3IXMEA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.596591276 (261h34m29.467674331s) [noise=true , address=145.79.227.45:4229 , piece_id=6FOOUZPUJFXPRNIJ4MUPCNFDDNBEIPEX372CSAL6OHOIAZJP4NBA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599217586 (261h34m29.470300621s) [noise=true , address=145.79.227.46:4601 , piece_id=GCJXJOBGRNMIHTTVRZWAX5UHWZOCC7PR3DFQGESGPI47GELKHLCQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.599865620 (261h34m29.470948665s) [noise=true , address=145.79.227.28:4727 , piece_id=L7SILI62ESVGSY3KHGTFPEPXWWCPNK5NZFAA2VAQNCHYN3XCEX4Q, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.601142182 (261h34m29.472225227s) [noise=true , address=145.79.227.2:4114 , piece_id=QSZFQM5F422KZEM5XMIITP5XTBFRE75ALEBQODQED2T2PG7ITXXQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.719001928 (261h34m29.590084963s) [noise=true , address=145.79.227.6:4259 , piece_id=U6E4JG67CZ4MFTKQI37NFGZELEMWWBSYL2IWNGGURSZQKASN6TIA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.721030170 (261h34m29.592113205s) [noise=true , address=145.79.227.20:4439 , piece_id=R3MDBLDOWUG2EO6LLL7LZCQMFE735AMWHUPLX2AG2CKKWKHLHEJA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.737244916 (261h34m29.608327961s) [noise=true , address=145.79.227.27:4691 , piece_id=5ZB2SI464OYLBCK7NYC44256OP5TYVRK6GDDFVVR7PJJ4RYDWCQQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.796723894 (261h34m29.667806929s) [noise=true , address=45.91.116.14:4189 , piece_id=5TCYGTU2454ABC2HILLK5T4O7BNWWRPDRF6IVXL2PRE7BB32GC6A, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898268494 (261h34m29.769351539s) [noise=true , address=45.91.116.6:4974 , piece_id=5H6O5EWUKOASBSGP76HU33WCYTDYQPP3XUPHS77KHEHPZNWY2ZYA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: <nil> Dec 16 13:03:29.898384775 (261h34m29.76946781s) [seg_pos={0 0}, upload=229345]: Segment reached optimal threshold of 13 pieces. Dec 16 13:03:29.898653948 (261h34m29.769736993s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 16 13:03:29.898663225 (261h34m29.76974626s) [noise=true , address=45.91.116.12:4227 , piece_id=LRMUOHXAKGXFHKJYVZHBBS5AIXWVMWQKJHXLNLGZHSQUV3V34YFQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898837377 (261h34m29.769920422s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:60838->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 13:03:29.898879627 (261h34m29.769962672s) [noise=true , address=77.90.31.16:4736 , piece_id=FRMHJJMQY6HWQGKOM4J63ZFREK4OXLAXU7KW4ODWT3MEC3V3HLZQ, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.898997452 (261h34m29.770080497s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:50604->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 16 13:03:29.899006409 (261h34m29.770089454s) [noise=true , address=77.90.31.21:4877 , piece_id=B4IPE5AJ2O2N23OBCZDCQPKPFTCRE6BNBPGNL6FK7YPADTXLN5BA, seg_pos={0 0}, upload=229345]: Long tail cancellation due to optimal threshold Dec 16 13:03:29.899028631 (261h34m29.770111676s) [seg_pos={0 0}, upload=229345]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 13:03:29.899109956 (261h34m29.770192991s) [upload=229345]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:03:29.899117520 (261h34m29.770200565s) [upload=229345]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:03:30.213746944 (261h34m30.084829979s) [upload=229345]: Done uploading segments... Dec 16 13:03:30.213767122 (261h34m30.084850157s) [upload=229345]: Upload finished. err: <nil> Duration: 30.00350464s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [X......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.XX.] duration: 5.001473916s trace: https://jaeger.dev.storj.io/trace/7ff8e411d100779b FAIL (2025-12-16 13:03:40 UTC) orbiter warning: context deadline exceeded Duration: 5.00150145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX............................X..X..........................] duration: 5.807252045s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 13:03:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.807278997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX............................X..X.........................] duration: 6.026241242s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 13:02:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 6.026266018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX............................X..X........................] duration: 5.874384177s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 13:02:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.874407181s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.XX] duration: 5.001953529s trace: https://jaeger.dev.storj.io/trace/7ff8e411d100779b FAIL (2025-12-16 13:02:34 UTC) orbiter warning: context deadline exceeded Duration: 5.001982705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX............................X..X.......................] duration: 5.769944386s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 13:01:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.76997209s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.X] duration: 5.000904393s trace: https://jaeger.dev.storj.io/trace/5c9137d786c20d1 FAIL (2025-12-16 13:01:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000929329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX............................X..X......................] duration: 5.799108648s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 13:01:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.799133296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX............................X..X.....................] duration: 5.846138121s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 13:01:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.846171405s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX.] duration: 4.161256303s trace: https://jaeger.dev.storj.io/trace/6e02f100aab3b05f FAIL (2025-12-16 13:00:50 UTC) orbiter warning: context deadline exceeded Duration: 4.161278284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [..XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXXX] duration: 5.001306786s trace: https://jaeger.dev.storj.io/trace/6e02f100aab3b05f FAIL (2025-12-16 13:00:29 UTC) orbiter warning: context deadline exceeded Duration: 5.001335691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XX............................X..X....................] duration: 7.516777937s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 13:00:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 7.516806823s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXXX] duration: 5.001579502s trace: https://jaeger.dev.storj.io/trace/6dfdd3ad6fad6544 FAIL (2025-12-16 12:59:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001605623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XX............................X..X...................] duration: 5.802495814s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:59:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.80252472s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXXX] duration: 5.001993495s trace: https://jaeger.dev.storj.io/trace/6029b7cfe0324a1 FAIL (2025-12-16 12:58:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002036517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XX............................X..X..................] duration: 6.036761572s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:58:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 6.036789815s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XXX] duration: 5.001422656s trace: https://jaeger.dev.storj.io/trace/69afc01ea39e0731 FAIL (2025-12-16 12:57:52 UTC) orbiter warning: context deadline exceeded Duration: 5.001454397s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [..X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..XX] duration: 5.001011804s trace: https://jaeger.dev.storj.io/trace/272991ee52aa5091 FAIL (2025-12-16 12:57:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001035069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XX............................X..X.................] duration: 6.097277888s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:57:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 6.09730483s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [...X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..X] duration: 5.001673946s trace: https://jaeger.dev.storj.io/trace/547f3eb3dcadb041 FAIL (2025-12-16 12:56:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001700057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XX............................X..X................] duration: 5.795976714s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:56:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.796006101s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [....X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX..] duration: 4.988205499s trace: https://jaeger.dev.storj.io/trace/2469bf9095be3926 FAIL (2025-12-16 12:55:55 UTC) orbiter warning: context deadline exceeded Duration: 4.988251207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XX............................X..X...............] duration: 5.778913223s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:55:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.778942198s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [X....X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX.] duration: 3.758468985s trace: https://jaeger.dev.storj.io/trace/2469bf9095be3926 FAIL (2025-12-16 12:54:58 UTC) orbiter warning: context deadline exceeded Duration: 3.758493201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XX............................X..X..............] duration: 5.823752333s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:54:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.82380392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XX............................X..X.............] duration: 5.795726196s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.795754308s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [.X....X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXXX] duration: 5.000735233s trace: https://jaeger.dev.storj.io/trace/2469bf9095be3926 FAIL (2025-12-16 12:53:53 UTC) orbiter warning: context deadline exceeded Duration: 5.000758967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX............................X..X............] duration: 5.74369614s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:53:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.743765031s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [..X....X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXXX] duration: 5.001613679s trace: https://jaeger.dev.storj.io/trace/7550f8c1dbc11cfe FAIL (2025-12-16 12:53:03 UTC) orbiter warning: context deadline exceeded Duration: 5.001642404s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [...X....X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXXX] duration: 5.001529479s trace: https://jaeger.dev.storj.io/trace/51fda4c2ebb90610 FAIL (2025-12-16 12:52:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001548657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX............................X..X...........] duration: 5.802684182s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:52:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.802712016s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [....X....X...XXX......X.X...X........X.XXXXXXXXXX.X..XX.XXXX] duration: 5.001380717s trace: https://jaeger.dev.storj.io/trace/341ff0a908ff8ae7 FAIL (2025-12-16 12:52:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00140841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XX............................X..X..........] duration: 5.781079258s trace: https://jaeger.dev.storj.io/trace/6da156ed54963bae FAIL (2025-12-16 12:52:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ) context deadline exceeded ======LOGS====== Dec 16 12:43:13.728557058 (261h14m13.599640103s) [upload=229047]: Starting upload Dec 16 12:43:13.728630357 (261h14m13.599713392s) [upload=229047]: Done starting upload Dec 16 12:43:13.728720429 (261h14m13.599803464s) [upload=229047]: Uploading segments... Dec 16 12:43:13.728870865 (261h14m13.59995391s) [upload=229047]: Got next segment. Inline: false Dec 16 12:43:13.728909358 (261h14m13.599992394s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 12:43:13.728916722 (261h14m13.599999757s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 12:43:13.972957534 (261h14m13.844040579s) [seg_pos={0 0}, upload=229047]: Begin upload segment... Dec 16 12:43:13.973018189 (261h14m13.844101224s) [seg_pos={0 0}, upload=229047]: Done begin upload segment. Dec 16 12:43:13.973047195 (261h14m13.84413023s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973077572 (261h14m13.844160607s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973129691 (261h14m13.844212727s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.973277132 (261h14m13.844360167s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974019627 (261h14m13.845102672s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974469202 (261h14m13.845552237s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974574993 (261h14m13.845658038s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.974906495 (261h14m13.84598954s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.975608862 (261h14m13.846691907s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.977027534 (261h14m13.848110579s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.978403523 (261h14m13.849486568s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.979495323 (261h14m13.850578368s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.980210935 (261h14m13.85129398s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.981410179 (261h14m13.852493214s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984172609 (261h14m13.855255654s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:13.984662782 (261h14m13.855745827s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Uploading piece... Dec 16 12:43:15.086530873 (261h14m14.957613918s) [noise=true , address=77.90.31.21:4877 , piece_id=HX5W43RBRV77ZVEW6PNPFJYNY42GHPP6IGM3EBYVRJOOULYTO6GQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.669980685 (261h14m15.54106373s) [upload=229047]: Next returned nil segment Dec 16 12:43:15.669997526 (261h14m15.541080571s) [upload=229047]: Waiting for error group managing segments... Dec 16 12:43:15.903820400 (261h14m15.774903435s) [noise=true , address=77.90.31.10:4517 , piece_id=LCWP7QIRJEIU6ZAPN7UENDAIH2AP6PSNJBUROBVDA53TQXXUMY6A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 12:43:15.951902859 (261h14m15.822985904s) [noise=true , address=145.79.227.10:4403 , piece_id=EXJQUKT3RRKVPYILF7OHM4KS65YBNJOBLZ5BDX4FQKZNPCZPLOXA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973291127 (261h14m15.844374162s) [noise=true , address=145.79.227.42:4123 , piece_id=FHBA5MWQDIO2AR2WSZKOFIOQ5CLU3KUUO6KZBI5GIYTPFSXMC6UA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.973360790 (261h14m15.844443835s) [noise=true , address=145.79.227.23:4545 , piece_id=CMBWXQ5JQ7VMLECNCREEBSYCNEIJBLD7VURTIWE3EOAJ437255JA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.982520812 (261h14m15.853603857s) [noise=true , address=145.79.227.38:4979 , piece_id=ATQJMSS4OGOYOV7JHJIR5L2AXJ3JG2BDCZSH2BSMFKTCLFWBRPQQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.984472408 (261h14m15.855555443s) [noise=true , address=145.79.227.49:4337 , piece_id=4EBKFKOE7QQ2H3XA6ZMYVZLEIAIQI3OVIIABVNBXMNXBMJO76TVQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986032358 (261h14m15.857115403s) [noise=true , address=145.79.227.15:4876 , piece_id=HTTRABRS77W63M247BPVM4355EXJ2XM5RAE4WYTU5Y57BW3KNTMA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.986742190 (261h14m15.857825225s) [noise=true , address=145.79.227.35:4483 , piece_id=MHTKY2G6SIU2PNBESDVHCXBXHZGFR7YPDN7J3YU4CJDF5BF3VMVA, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:15.996623816 (261h14m15.867706851s) [noise=true , address=145.79.227.45:4230 , piece_id=OXTKOHXIDLFHNWB6AGZTO46W6ISVECHVXTUMB36JC6JICYQAHPNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.051285142 (261h14m15.922368178s) [noise=true , address=145.79.227.36:4519 , piece_id=3PCQPIRPQRYNQLQ3E26DSPFEAN5OIZSPWLFNJFXG7LGSL7R5R56Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.055795661 (261h14m15.926878696s) [noise=true , address=145.79.227.37:4567 , piece_id=BKXYONQBJCTHDQTOCIU4EL5DHIBFHG72ITE7DHULTGWQAOL4TGNQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.292043703 (261h14m16.163126758s) [noise=true , address=45.91.116.14:4190 , piece_id=CQDXY75T4WKG2LOAARKP6M4UBZ7ESH3YY2Y6TZR23OFQQBBSCK7A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:16.323293028 (261h14m16.194376073s) [noise=true , address=45.91.116.8:4045 , piece_id=S5CJERIQY3GZAJXN6ZO4F7KPWG6YRIEUHZUVG32G6F2AD2IB7J6Q, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080613814 (261h14m16.951696859s) [noise=true , address=45.91.116.10:4084 , piece_id=PIM2A7BAW5JLLAUXC5KY3OIJVDUFMD2QJOMAWO2YZPDGYTV3O5UQ, seg_pos={0 0}, upload=229047]: Done uploading piece. err: <nil> Dec 16 12:43:17.080636187 (261h14m16.951719222s) [seg_pos={0 0}, upload=229047]: Segment reached optimal threshold of 13 pieces. Dec 16 12:43:17.080908635 (261h14m16.95199169s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 16 12:43:17.080919286 (261h14m16.952002321s) [noise=true , address=145.79.227.30:4762 , piece_id=43BVWF67M2D2JGE6GYLEHCNG2AWDSZAWHJD4MLUXK5SYCNY7RV2A, seg_pos={0 0}, upload=229047]: Long tail cancellation due to optimal threshold Dec 16 12:43:17.080942920 (261h14m16.952025955s) [seg_pos={0 0}, upload=229047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 12:43:17.081036538 (261h14m16.952119583s) [upload=229047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 12:43:17.081043632 (261h14m16.952126677s) [upload=229047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 12:43:17.371280229 (261h14m17.242363264s) [upload=229047]: Done uploading segments... Dec 16 12:43:17.371312330 (261h14m17.242395375s) [upload=229047]: Upload finished. err: <nil> Duration: 5.781109384s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)