Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.816119492s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 17:48:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.816156303s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 4.827794281s trace: https://jaeger.dev.storj.io/trace/7058122c2f3fc553 FAIL (2025-12-08 17:48:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.827824719s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.591062461s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 17:48:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.591195654s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.417897945s trace: https://jaeger.dev.storj.io/trace/136ff0f218b6bc4c FAIL (2025-12-08 17:47:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417926419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.692283606s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 17:47:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.692312289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.036274376s trace: https://jaeger.dev.storj.io/trace/71acc8d1f58c5156 FAIL (2025-12-08 17:47:37 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:19:35.057390695 (73h50m34.92847375s) [upload=64625]: Starting upload Dec 8 17:19:35.057451541 (73h50m34.928534586s) [upload=64625]: Done starting upload Dec 8 17:19:35.057525381 (73h50m34.928608426s) [upload=64625]: Uploading segments... Dec 8 17:19:35.057592810 (73h50m34.928675855s) [upload=64625]: Got next segment. Inline: true Dec 8 17:19:35.057598100 (73h50m34.928681145s) [upload=64625]: Waiting for error group managing segments... Dec 8 17:19:35.057643065 (73h50m34.92872611s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:19:35.057648215 (73h50m34.92873126s) [upload=64625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 17:19:35.057651671 (73h50m34.928734716s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:19:35.334831026 (73h50m35.205914061s) [upload=64625]: Done uploading segments... Dec 8 17:19:35.334853018 (73h50m35.205936063s) [upload=64625]: Upload finished. err: <nil> Duration:* 2.036401559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.503421917s trace: https://jaeger.dev.storj.io/trace/7058122c2f3fc553 FAIL (2025-12-08 17:47:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.503461472s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.375775968s trace: https://jaeger.dev.storj.io/trace/136ff0f218b6bc4c FAIL (2025-12-08 17:47:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.375804101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.99925264s trace: https://jaeger.dev.storj.io/trace/46021ff75bd7decc FAIL (2025-12-08 17:47:24 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.999280634s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.016720535s trace: https://jaeger.dev.storj.io/trace/71acc8d1f58c5156 FAIL (2025-12-08 17:47:13 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:19:35.057390695 (73h50m34.92847375s) [upload=64625]: Starting upload Dec 8 17:19:35.057451541 (73h50m34.928534586s) [upload=64625]: Done starting upload Dec 8 17:19:35.057525381 (73h50m34.928608426s) [upload=64625]: Uploading segments... Dec 8 17:19:35.057592810 (73h50m34.928675855s) [upload=64625]: Got next segment. Inline: true Dec 8 17:19:35.057598100 (73h50m34.928681145s) [upload=64625]: Waiting for error group managing segments... Dec 8 17:19:35.057643065 (73h50m34.92872611s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:19:35.057648215 (73h50m34.92873126s) [upload=64625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 17:19:35.057651671 (73h50m34.928734716s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:19:35.334831026 (73h50m35.205914061s) [upload=64625]: Done uploading segments... Dec 8 17:19:35.334853018 (73h50m35.205936063s) [upload=64625]: Upload finished. err: <nil> Duration:* 2.016748538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.855772172s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 17:47:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.855803422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.836701267s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 17:47:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.836731114s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.489897581s trace: https://jaeger.dev.storj.io/trace/136ff0f218b6bc4c FAIL (2025-12-08 17:47:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.489921006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 28.251500406s trace: https://jaeger.dev.storj.io/trace/7058122c2f3fc553 FAIL (2025-12-08 17:46:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 28.251532409s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.783484918s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 17:46:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.783514805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.039410741s trace: https://jaeger.dev.storj.io/trace/46021ff75bd7decc FAIL (2025-12-08 17:46:52 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.039437543s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.038379597s trace: https://jaeger.dev.storj.io/trace/71acc8d1f58c5156 FAIL (2025-12-08 17:46:51 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:19:35.057390695 (73h50m34.92847375s) [upload=64625]: Starting upload Dec 8 17:19:35.057451541 (73h50m34.928534586s) [upload=64625]: Done starting upload Dec 8 17:19:35.057525381 (73h50m34.928608426s) [upload=64625]: Uploading segments... Dec 8 17:19:35.057592810 (73h50m34.928675855s) [upload=64625]: Got next segment. Inline: true Dec 8 17:19:35.057598100 (73h50m34.928681145s) [upload=64625]: Waiting for error group managing segments... Dec 8 17:19:35.057643065 (73h50m34.92872611s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:19:35.057648215 (73h50m34.92873126s) [upload=64625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 17:19:35.057651671 (73h50m34.928734716s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:19:35.334831026 (73h50m35.205914061s) [upload=64625]: Done uploading segments... Dec 8 17:19:35.334853018 (73h50m35.205936063s) [upload=64625]: Upload finished. err: <nil> Duration:* 2.038405346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.892873619s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 17:46:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.892943481s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.01450234s trace: https://jaeger.dev.storj.io/trace/71acc8d1f58c5156 FAIL (2025-12-08 17:46:28 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:19:35.057390695 (73h50m34.92847375s) [upload=64625]: Starting upload Dec 8 17:19:35.057451541 (73h50m34.928534586s) [upload=64625]: Done starting upload Dec 8 17:19:35.057525381 (73h50m34.928608426s) [upload=64625]: Uploading segments... Dec 8 17:19:35.057592810 (73h50m34.928675855s) [upload=64625]: Got next segment. Inline: true Dec 8 17:19:35.057598100 (73h50m34.928681145s) [upload=64625]: Waiting for error group managing segments... Dec 8 17:19:35.057643065 (73h50m34.92872611s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:19:35.057648215 (73h50m34.92873126s) [upload=64625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 17:19:35.057651671 (73h50m34.928734716s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:19:35.334831026 (73h50m35.205914061s) [upload=64625]: Done uploading segments... Dec 8 17:19:35.334853018 (73h50m35.205936063s) [upload=64625]: Upload finished. err: <nil> Duration:* 2.014530814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.........X......................X............] duration: 5.783063989s trace: https://jaeger.dev.storj.io/trace/7f2371936424c2ff FAIL (2025-12-08 17:46:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.783094888s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.07271575s trace: https://jaeger.dev.storj.io/trace/46021ff75bd7decc FAIL (2025-12-08 17:46:08 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.072748451s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.814106708s trace: https://jaeger.dev.storj.io/trace/7058122c2f3fc553 FAIL (2025-12-08 17:45:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.81414357s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.015080558s trace: https://jaeger.dev.storj.io/trace/71acc8d1f58c5156 FAIL (2025-12-08 17:45:56 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:19:35.057390695 (73h50m34.92847375s) [upload=64625]: Starting upload Dec 8 17:19:35.057451541 (73h50m34.928534586s) [upload=64625]: Done starting upload Dec 8 17:19:35.057525381 (73h50m34.928608426s) [upload=64625]: Uploading segments... Dec 8 17:19:35.057592810 (73h50m34.928675855s) [upload=64625]: Got next segment. Inline: true Dec 8 17:19:35.057598100 (73h50m34.928681145s) [upload=64625]: Waiting for error group managing segments... Dec 8 17:19:35.057643065 (73h50m34.92872611s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:19:35.057648215 (73h50m34.92873126s) [upload=64625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 17:19:35.057651671 (73h50m34.928734716s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:19:35.334831026 (73h50m35.205914061s) [upload=64625]: Done uploading segments... Dec 8 17:19:35.334853018 (73h50m35.205936063s) [upload=64625]: Upload finished. err: <nil> Duration:* 2.015098403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.536080019s trace: https://jaeger.dev.storj.io/trace/136ff0f218b6bc4c FAIL (2025-12-08 17:45:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.536131175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.716022921s trace: https://jaeger.dev.storj.io/trace/2bacc06da04f4505 FAIL (2025-12-08 17:45:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=185.167.204.206:4049 , piece_id=2NHQKCEPOXX475AS5SIZ7IXCNKJMXWW6LNU2OO6PBILRZ6DUZ5EA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.188203846 (73h32m28.059286901s) [noise=true , address=185.167.204.206:4069 , piece_id=XGOYKYXSUAK4IHEM44RZB4DHNXYXIYQTVWZDNAPNE5G5HWUK4VFA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.189671170 (73h32m28.060754215s) [noise=true , address=185.167.204.206:4050 , piece_id=BETXF3XMM5F42JB44YF2QGK6N54JOA6CZ4Z2EV5WOXZPUD2KKKTA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.191039326 (73h32m28.062122361s) [noise=true , address=81.171.14.70:4044 , piece_id=I4DPAS3D7CK5K5RMS7KASK775IF2QTC5SLDRNEXERI7CPHACCSMA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.193557270 (73h32m28.064640305s) [noise=true , address=185.167.204.206:4061 , piece_id=Y6BICZVIUK6OFLRXZJE3S4VVFIRWSAIJBAFOL6AEY66CCKZQFCTQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.195856628 (73h32m28.066939674s) [noise=true , address=185.167.204.205:4015 , piece_id=FJKFMAGYH277I25J3WQ3IWR3DYRWICWQRC5LSKDDJ63XZ74S7MXQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.198414219 (73h32m28.069497264s) [noise=true , address=185.167.204.205:4030 , piece_id=I5Y6S4GGMJK3HALRUWMSKQVLZMCSD5Q34GHLOJDKEOBWY7KLRMTQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.201095685 (73h32m28.07217873s) [noise=true , address=81.171.14.70:4032 , piece_id=ZMDAJH2OX64GP4YR5E7CBNJC72RDTRMTCXEW67KIGQIZHJBO3ECQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.204972177 (73h32m28.076055212s) [noise=true , address=185.167.204.206:4064 , piece_id=NCWE447MXDE4SAN7MSMHMIU2TCDH2X7M5OJSICXKCE2Y3OFEACZA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:29.014501224 (73h32m28.885584269s) [upload=64365]: Next returned nil segment Dec 8 17:01:29.014527022 (73h32m28.885610067s) [upload=64365]: Waiting for error group managing segments... Dec 8 17:01:29.408440684 (73h32m29.279523729s) [noise=true , address=185.167.204.205:4023 , piece_id=V344WVXRV2SEML6TEIASWYT4427PRE56R3KLFQJZ4ZNUFCQTNF7Q, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.409778682 (73h32m29.280861728s) [noise=true , address=185.167.204.205:4034 , piece_id=OGTA4AK7H6XFMPSB2HDYAYMVUHLMFU5IYM3TKOFWWDZBHYFAAESQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.411240316 (73h32m29.282323361s) [noise=true , address=185.167.204.206:4053 , piece_id=4MMKVKWPWU6CK5M7R74PXGQ34PPUXGWUJT2LJ6ZEXV6A2X72PJEQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.412578493 (73h32m29.283661538s) [noise=true , address=185.167.204.205:4036 , piece_id=6VYQ5BV7BKIGEIHKBERSBYRKTP7VC7M6NPNKWRSYSRVP243OVV4A, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.419000773 (73h32m29.290083818s) [noise=true , address=185.167.204.205:4029 , piece_id=KNEK65KYMS5VO2REHA4UZ6WXFVKYT7LUQ47WPWDH6U6IWBYYFFXQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.427501339 (73h32m29.298584384s) [noise=true , address=185.167.204.206:4068 , piece_id=4ZBFLMHY2SZWQKT6SLGWH67EYB2UYOBVFW32X3NMEKJ2QGHUB62Q, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.431109811 (73h32m29.302192856s) [noise=true , address=185.167.204.205:4030 , piece_id=I5Y6S4GGMJK3HALRUWMSKQVLZMCSD5Q34GHLOJDKEOBWY7KLRMTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.432493316 (73h32m29.303576351s) [noise=true , address=185.167.204.206:4037 , piece_id=JWENQ4SQJCAD6RAEM35TQZNDXTMUOGV7JGWOLGA64BFQUF2MDKQQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.432695100 (73h32m29.303778135s) [noise=true , address=185.167.204.206:4070 , piece_id=RIILUOUAANPXHFPC67DDMBBXDRCIPMNGJKHPSGKBWPIKZLLB3SIQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.433119147 (73h32m29.304202192s) [noise=true , address=185.167.204.206:4039 , piece_id=56TVAHFZJLIWEWOWZCLQML2LX4IDFAERQQXRHDI2WAQPDDS36YNA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.434343859 (73h32m29.305426904s) [noise=true , address=185.167.204.206:4061 , piece_id=Y6BICZVIUK6OFLRXZJE3S4VVFIRWSAIJBAFOL6AEY66CCKZQFCTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.434887174 (73h32m29.305970209s) [noise=true , address=185.167.204.206:4064 , piece_id=NCWE447MXDE4SAN7MSMHMIU2TCDH2X7M5OJSICXKCE2Y3OFEACZA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.439186210 (73h32m29.310269255s) [noise=true , address=185.167.204.206:4050 , piece_id=BETXF3XMM5F42JB44YF2QGK6N54JOA6CZ4Z2EV5WOXZPUD2KKKTA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.439715649 (73h32m29.310798684s) [noise=true , address=185.167.204.206:4069 , piece_id=XGOYKYXSUAK4IHEM44RZB4DHNXYXIYQTVWZDNAPNE5G5HWUK4VFA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.440245988 (73h32m29.311329033s) [noise=true , address=185.167.204.205:4001 , piece_id=XAESPYQ55LBDJQ7WJIFWCVTCAGNXWAOOMSL3VMRERS3LZTMZB3GQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.440484813 (73h32m29.311567848s) [noise=true , address=185.167.204.206:4049 , piece_id=2NHQKCEPOXX475AS5SIZ7IXCNKJMXWW6LNU2OO6PBILRZ6DUZ5EA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.442183288 (73h32m29.313266323s) [noise=true , address=185.167.204.205:4015 , piece_id=FJKFMAGYH277I25J3WQ3IWR3DYRWICWQRC5LSKDDJ63XZ74S7MXQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.444441307 (73h32m29.315524352s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDPLG5LX3S7NJZDZSB4NQTMY4KBHBN3N5UGREBDB5JJHKBW3WWQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.445075635 (73h32m29.31615868s) [noise=true , address=185.167.204.205:4025 , piece_id=6VYO47IF26LZ5LO7KSUMRFMWOL6PKBOGT75UQQGCQUYX6CEFOVIQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.448313100 (73h32m29.319396135s) [noise=true , address=185.167.204.206:4048 , piece_id=BU4IJLUUZ66CLPARWCCMRZCCDQYHYWZOGK2EASU3HRNWLQ7B5RUA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.450008007 (73h32m29.321091042s) [noise=true , address=185.167.204.206:4057 , piece_id=ZLIWTLFQD4KDPMWOKRZUAVTK4BUF2J3BH6ETCMWBC7HMN5JBUQJA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.464765288 (73h32m29.335848323s) [noise=true , address=185.167.204.205:4007 , piece_id=45SASPPJTGKEBQTAINBPGKBUCTPJH4QXLGP34TK5H6MF5BK5AUAA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.006798349 (73h32m29.877881394s) [noise=true , address=81.171.14.70:4044 , piece_id=I4DPAS3D7CK5K5RMS7KASK775IF2QTC5SLDRNEXERI7CPHACCSMA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.052457282 (73h32m29.923540327s) [noise=true , address=81.171.14.70:4026 , piece_id=KL7AYBMWQ5P63JLIQPHVHBVZOFTOMSS2XJF2QTTIRQGF7ROTYPTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.087401313 (73h32m29.958484358s) [noise=true , address=81.171.14.70:4027 , piece_id=3XX2VTW3KPECPWINTSTFFUQL5CRKO4GENBS2TVTNSIN44YXCETJA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.150558347 (73h32m30.021641392s) [noise=true , address=81.171.14.70:4032 , piece_id=ZMDAJH2OX64GP4YR5E7CBNJC72RDTRMTCXEW67KIGQIZHJBO3ECQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.150583825 (73h32m30.02166687s) [seg_pos={0 0}, upload=64365]: Segment reached optimal threshold of 26 pieces. Dec 8 17:01:30.150889357 (73h32m30.021972412s) [noise=true , address=89.149.201.147:4016 , piece_id=BQT6TNPFUVT2OTT4YIRIC3CNAPIANWHIT23W5JPPLK6Y542ZER7A, seg_pos={0 0}, upload=64365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection; write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection Dec 8 17:01:30.150901891 (73h32m30.021984936s) [noise=true , address=89.149.201.147:4016 , piece_id=BQT6TNPFUVT2OTT4YIRIC3CNAPIANWHIT23W5JPPLK6Y542ZER7A, seg_pos={0 0}, upload=64365]: Long tail cancellation due to optimal threshold Dec 8 17:01:30.150914835 (73h32m30.02199788s) [noise=true , address=89.149.201.147:4011 , piece_id=BEU4ZFPKV5CFS3EU5FL5IOBSR2462IN5SUPKAEAWPGAHXWELR5AA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: read tcp 188.166.225.170:55962->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:01:30.150922571 (73h32m30.022005626s) [noise=true , address=89.149.201.147:4011 , piece_id=BEU4ZFPKV5CFS3EU5FL5IOBSR2462IN5SUPKAEAWPGAHXWELR5AA, seg_pos={0 0}, upload=64365]: Long tail cancellation due to optimal threshold Dec 8 17:01:30.150944682 (73h32m30.022027727s) [seg_pos={0 0}, upload=64365]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 17:01:30.151058490 (73h32m30.022141535s) [upload=64365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:01:30.151066825 (73h32m30.02214987s) [upload=64365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:03:21.289875989 (73h34m21.160959045s) [upload=64365]: Done uploading segments... Dec 8 17:03:21.289984466 (73h34m21.161067531s) [upload=64365]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.716052106s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.960535962s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 17:45:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.960566481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.805483581s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 17:45:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.805512056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.037075703s trace: https://jaeger.dev.storj.io/trace/46021ff75bd7decc FAIL (2025-12-08 17:45:25 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.037108716s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.036569059s trace: https://jaeger.dev.storj.io/trace/71acc8d1f58c5156 FAIL (2025-12-08 17:45:15 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:19:35.057390695 (73h50m34.92847375s) [upload=64625]: Starting upload Dec 8 17:19:35.057451541 (73h50m34.928534586s) [upload=64625]: Done starting upload Dec 8 17:19:35.057525381 (73h50m34.928608426s) [upload=64625]: Uploading segments... Dec 8 17:19:35.057592810 (73h50m34.928675855s) [upload=64625]: Got next segment. Inline: true Dec 8 17:19:35.057598100 (73h50m34.928681145s) [upload=64625]: Waiting for error group managing segments... Dec 8 17:19:35.057643065 (73h50m34.92872611s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:19:35.057648215 (73h50m34.92873126s) [upload=64625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 17:19:35.057651671 (73h50m34.928734716s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:19:35.334831026 (73h50m35.205914061s) [upload=64625]: Done uploading segments... Dec 8 17:19:35.334853018 (73h50m35.205936063s) [upload=64625]: Upload finished. err: <nil> Duration:* 2.036598385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.541842645s trace: https://jaeger.dev.storj.io/trace/2bacc06da04f4505 FAIL (2025-12-08 17:45:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=185.167.204.206:4049 , piece_id=2NHQKCEPOXX475AS5SIZ7IXCNKJMXWW6LNU2OO6PBILRZ6DUZ5EA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.188203846 (73h32m28.059286901s) [noise=true , address=185.167.204.206:4069 , piece_id=XGOYKYXSUAK4IHEM44RZB4DHNXYXIYQTVWZDNAPNE5G5HWUK4VFA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.189671170 (73h32m28.060754215s) [noise=true , address=185.167.204.206:4050 , piece_id=BETXF3XMM5F42JB44YF2QGK6N54JOA6CZ4Z2EV5WOXZPUD2KKKTA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.191039326 (73h32m28.062122361s) [noise=true , address=81.171.14.70:4044 , piece_id=I4DPAS3D7CK5K5RMS7KASK775IF2QTC5SLDRNEXERI7CPHACCSMA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.193557270 (73h32m28.064640305s) [noise=true , address=185.167.204.206:4061 , piece_id=Y6BICZVIUK6OFLRXZJE3S4VVFIRWSAIJBAFOL6AEY66CCKZQFCTQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.195856628 (73h32m28.066939674s) [noise=true , address=185.167.204.205:4015 , piece_id=FJKFMAGYH277I25J3WQ3IWR3DYRWICWQRC5LSKDDJ63XZ74S7MXQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.198414219 (73h32m28.069497264s) [noise=true , address=185.167.204.205:4030 , piece_id=I5Y6S4GGMJK3HALRUWMSKQVLZMCSD5Q34GHLOJDKEOBWY7KLRMTQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.201095685 (73h32m28.07217873s) [noise=true , address=81.171.14.70:4032 , piece_id=ZMDAJH2OX64GP4YR5E7CBNJC72RDTRMTCXEW67KIGQIZHJBO3ECQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.204972177 (73h32m28.076055212s) [noise=true , address=185.167.204.206:4064 , piece_id=NCWE447MXDE4SAN7MSMHMIU2TCDH2X7M5OJSICXKCE2Y3OFEACZA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:29.014501224 (73h32m28.885584269s) [upload=64365]: Next returned nil segment Dec 8 17:01:29.014527022 (73h32m28.885610067s) [upload=64365]: Waiting for error group managing segments... Dec 8 17:01:29.408440684 (73h32m29.279523729s) [noise=true , address=185.167.204.205:4023 , piece_id=V344WVXRV2SEML6TEIASWYT4427PRE56R3KLFQJZ4ZNUFCQTNF7Q, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.409778682 (73h32m29.280861728s) [noise=true , address=185.167.204.205:4034 , piece_id=OGTA4AK7H6XFMPSB2HDYAYMVUHLMFU5IYM3TKOFWWDZBHYFAAESQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.411240316 (73h32m29.282323361s) [noise=true , address=185.167.204.206:4053 , piece_id=4MMKVKWPWU6CK5M7R74PXGQ34PPUXGWUJT2LJ6ZEXV6A2X72PJEQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.412578493 (73h32m29.283661538s) [noise=true , address=185.167.204.205:4036 , piece_id=6VYQ5BV7BKIGEIHKBERSBYRKTP7VC7M6NPNKWRSYSRVP243OVV4A, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.419000773 (73h32m29.290083818s) [noise=true , address=185.167.204.205:4029 , piece_id=KNEK65KYMS5VO2REHA4UZ6WXFVKYT7LUQ47WPWDH6U6IWBYYFFXQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.427501339 (73h32m29.298584384s) [noise=true , address=185.167.204.206:4068 , piece_id=4ZBFLMHY2SZWQKT6SLGWH67EYB2UYOBVFW32X3NMEKJ2QGHUB62Q, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.431109811 (73h32m29.302192856s) [noise=true , address=185.167.204.205:4030 , piece_id=I5Y6S4GGMJK3HALRUWMSKQVLZMCSD5Q34GHLOJDKEOBWY7KLRMTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.432493316 (73h32m29.303576351s) [noise=true , address=185.167.204.206:4037 , piece_id=JWENQ4SQJCAD6RAEM35TQZNDXTMUOGV7JGWOLGA64BFQUF2MDKQQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.432695100 (73h32m29.303778135s) [noise=true , address=185.167.204.206:4070 , piece_id=RIILUOUAANPXHFPC67DDMBBXDRCIPMNGJKHPSGKBWPIKZLLB3SIQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.433119147 (73h32m29.304202192s) [noise=true , address=185.167.204.206:4039 , piece_id=56TVAHFZJLIWEWOWZCLQML2LX4IDFAERQQXRHDI2WAQPDDS36YNA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.434343859 (73h32m29.305426904s) [noise=true , address=185.167.204.206:4061 , piece_id=Y6BICZVIUK6OFLRXZJE3S4VVFIRWSAIJBAFOL6AEY66CCKZQFCTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.434887174 (73h32m29.305970209s) [noise=true , address=185.167.204.206:4064 , piece_id=NCWE447MXDE4SAN7MSMHMIU2TCDH2X7M5OJSICXKCE2Y3OFEACZA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.439186210 (73h32m29.310269255s) [noise=true , address=185.167.204.206:4050 , piece_id=BETXF3XMM5F42JB44YF2QGK6N54JOA6CZ4Z2EV5WOXZPUD2KKKTA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.439715649 (73h32m29.310798684s) [noise=true , address=185.167.204.206:4069 , piece_id=XGOYKYXSUAK4IHEM44RZB4DHNXYXIYQTVWZDNAPNE5G5HWUK4VFA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.440245988 (73h32m29.311329033s) [noise=true , address=185.167.204.205:4001 , piece_id=XAESPYQ55LBDJQ7WJIFWCVTCAGNXWAOOMSL3VMRERS3LZTMZB3GQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.440484813 (73h32m29.311567848s) [noise=true , address=185.167.204.206:4049 , piece_id=2NHQKCEPOXX475AS5SIZ7IXCNKJMXWW6LNU2OO6PBILRZ6DUZ5EA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.442183288 (73h32m29.313266323s) [noise=true , address=185.167.204.205:4015 , piece_id=FJKFMAGYH277I25J3WQ3IWR3DYRWICWQRC5LSKDDJ63XZ74S7MXQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.444441307 (73h32m29.315524352s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDPLG5LX3S7NJZDZSB4NQTMY4KBHBN3N5UGREBDB5JJHKBW3WWQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.445075635 (73h32m29.31615868s) [noise=true , address=185.167.204.205:4025 , piece_id=6VYO47IF26LZ5LO7KSUMRFMWOL6PKBOGT75UQQGCQUYX6CEFOVIQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.448313100 (73h32m29.319396135s) [noise=true , address=185.167.204.206:4048 , piece_id=BU4IJLUUZ66CLPARWCCMRZCCDQYHYWZOGK2EASU3HRNWLQ7B5RUA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.450008007 (73h32m29.321091042s) [noise=true , address=185.167.204.206:4057 , piece_id=ZLIWTLFQD4KDPMWOKRZUAVTK4BUF2J3BH6ETCMWBC7HMN5JBUQJA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.464765288 (73h32m29.335848323s) [noise=true , address=185.167.204.205:4007 , piece_id=45SASPPJTGKEBQTAINBPGKBUCTPJH4QXLGP34TK5H6MF5BK5AUAA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.006798349 (73h32m29.877881394s) [noise=true , address=81.171.14.70:4044 , piece_id=I4DPAS3D7CK5K5RMS7KASK775IF2QTC5SLDRNEXERI7CPHACCSMA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.052457282 (73h32m29.923540327s) [noise=true , address=81.171.14.70:4026 , piece_id=KL7AYBMWQ5P63JLIQPHVHBVZOFTOMSS2XJF2QTTIRQGF7ROTYPTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.087401313 (73h32m29.958484358s) [noise=true , address=81.171.14.70:4027 , piece_id=3XX2VTW3KPECPWINTSTFFUQL5CRKO4GENBS2TVTNSIN44YXCETJA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.150558347 (73h32m30.021641392s) [noise=true , address=81.171.14.70:4032 , piece_id=ZMDAJH2OX64GP4YR5E7CBNJC72RDTRMTCXEW67KIGQIZHJBO3ECQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.150583825 (73h32m30.02166687s) [seg_pos={0 0}, upload=64365]: Segment reached optimal threshold of 26 pieces. Dec 8 17:01:30.150889357 (73h32m30.021972412s) [noise=true , address=89.149.201.147:4016 , piece_id=BQT6TNPFUVT2OTT4YIRIC3CNAPIANWHIT23W5JPPLK6Y542ZER7A, seg_pos={0 0}, upload=64365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection; write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection Dec 8 17:01:30.150901891 (73h32m30.021984936s) [noise=true , address=89.149.201.147:4016 , piece_id=BQT6TNPFUVT2OTT4YIRIC3CNAPIANWHIT23W5JPPLK6Y542ZER7A, seg_pos={0 0}, upload=64365]: Long tail cancellation due to optimal threshold Dec 8 17:01:30.150914835 (73h32m30.02199788s) [noise=true , address=89.149.201.147:4011 , piece_id=BEU4ZFPKV5CFS3EU5FL5IOBSR2462IN5SUPKAEAWPGAHXWELR5AA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: read tcp 188.166.225.170:55962->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:01:30.150922571 (73h32m30.022005626s) [noise=true , address=89.149.201.147:4011 , piece_id=BEU4ZFPKV5CFS3EU5FL5IOBSR2462IN5SUPKAEAWPGAHXWELR5AA, seg_pos={0 0}, upload=64365]: Long tail cancellation due to optimal threshold Dec 8 17:01:30.150944682 (73h32m30.022027727s) [seg_pos={0 0}, upload=64365]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 17:01:30.151058490 (73h32m30.022141535s) [upload=64365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:01:30.151066825 (73h32m30.02214987s) [upload=64365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:03:21.289875989 (73h34m21.160959045s) [upload=64365]: Done uploading segments... Dec 8 17:03:21.289984466 (73h34m21.161067531s) [upload=64365]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.541873375s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.........X......................X...........] duration: 6.112513801s trace: https://jaeger.dev.storj.io/trace/7f2371936424c2ff FAIL (2025-12-08 17:45:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.112546474s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.869145127s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 17:45:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.869177959s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.51826739s trace: https://jaeger.dev.storj.io/trace/136ff0f218b6bc4c FAIL (2025-12-08 17:45:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.518305163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.930248216s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 17:44:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.930278243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.027943397s trace: https://jaeger.dev.storj.io/trace/7058122c2f3fc553 FAIL (2025-12-08 17:44:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.027970971s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.020447128s trace: https://jaeger.dev.storj.io/trace/71acc8d1f58c5156 FAIL (2025-12-08 17:44:32 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:19:35.057390695 (73h50m34.92847375s) [upload=64625]: Starting upload Dec 8 17:19:35.057451541 (73h50m34.928534586s) [upload=64625]: Done starting upload Dec 8 17:19:35.057525381 (73h50m34.928608426s) [upload=64625]: Uploading segments... Dec 8 17:19:35.057592810 (73h50m34.928675855s) [upload=64625]: Got next segment. Inline: true Dec 8 17:19:35.057598100 (73h50m34.928681145s) [upload=64625]: Waiting for error group managing segments... Dec 8 17:19:35.057643065 (73h50m34.92872611s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:19:35.057648215 (73h50m34.92873126s) [upload=64625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 17:19:35.057651671 (73h50m34.928734716s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:19:35.334831026 (73h50m35.205914061s) [upload=64625]: Done uploading segments... Dec 8 17:19:35.334853018 (73h50m35.205936063s) [upload=64625]: Upload finished. err: <nil> Duration:* 2.020478739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.744221733s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 17:44:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.744241992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.........X......................X..........] duration: 6.093022147s trace: https://jaeger.dev.storj.io/trace/7f2371936424c2ff FAIL (2025-12-08 17:44:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.093050321s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.999602142s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 17:44:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.999629635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.030216573s trace: https://jaeger.dev.storj.io/trace/46021ff75bd7decc FAIL (2025-12-08 17:44:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.030244976s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 4.587690237s trace: https://jaeger.dev.storj.io/trace/2bacc06da04f4505 FAIL (2025-12-08 17:44:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=185.167.204.206:4049 , piece_id=2NHQKCEPOXX475AS5SIZ7IXCNKJMXWW6LNU2OO6PBILRZ6DUZ5EA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.188203846 (73h32m28.059286901s) [noise=true , address=185.167.204.206:4069 , piece_id=XGOYKYXSUAK4IHEM44RZB4DHNXYXIYQTVWZDNAPNE5G5HWUK4VFA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.189671170 (73h32m28.060754215s) [noise=true , address=185.167.204.206:4050 , piece_id=BETXF3XMM5F42JB44YF2QGK6N54JOA6CZ4Z2EV5WOXZPUD2KKKTA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.191039326 (73h32m28.062122361s) [noise=true , address=81.171.14.70:4044 , piece_id=I4DPAS3D7CK5K5RMS7KASK775IF2QTC5SLDRNEXERI7CPHACCSMA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.193557270 (73h32m28.064640305s) [noise=true , address=185.167.204.206:4061 , piece_id=Y6BICZVIUK6OFLRXZJE3S4VVFIRWSAIJBAFOL6AEY66CCKZQFCTQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.195856628 (73h32m28.066939674s) [noise=true , address=185.167.204.205:4015 , piece_id=FJKFMAGYH277I25J3WQ3IWR3DYRWICWQRC5LSKDDJ63XZ74S7MXQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.198414219 (73h32m28.069497264s) [noise=true , address=185.167.204.205:4030 , piece_id=I5Y6S4GGMJK3HALRUWMSKQVLZMCSD5Q34GHLOJDKEOBWY7KLRMTQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.201095685 (73h32m28.07217873s) [noise=true , address=81.171.14.70:4032 , piece_id=ZMDAJH2OX64GP4YR5E7CBNJC72RDTRMTCXEW67KIGQIZHJBO3ECQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.204972177 (73h32m28.076055212s) [noise=true , address=185.167.204.206:4064 , piece_id=NCWE447MXDE4SAN7MSMHMIU2TCDH2X7M5OJSICXKCE2Y3OFEACZA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:29.014501224 (73h32m28.885584269s) [upload=64365]: Next returned nil segment Dec 8 17:01:29.014527022 (73h32m28.885610067s) [upload=64365]: Waiting for error group managing segments... Dec 8 17:01:29.408440684 (73h32m29.279523729s) [noise=true , address=185.167.204.205:4023 , piece_id=V344WVXRV2SEML6TEIASWYT4427PRE56R3KLFQJZ4ZNUFCQTNF7Q, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.409778682 (73h32m29.280861728s) [noise=true , address=185.167.204.205:4034 , piece_id=OGTA4AK7H6XFMPSB2HDYAYMVUHLMFU5IYM3TKOFWWDZBHYFAAESQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.411240316 (73h32m29.282323361s) [noise=true , address=185.167.204.206:4053 , piece_id=4MMKVKWPWU6CK5M7R74PXGQ34PPUXGWUJT2LJ6ZEXV6A2X72PJEQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.412578493 (73h32m29.283661538s) [noise=true , address=185.167.204.205:4036 , piece_id=6VYQ5BV7BKIGEIHKBERSBYRKTP7VC7M6NPNKWRSYSRVP243OVV4A, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.419000773 (73h32m29.290083818s) [noise=true , address=185.167.204.205:4029 , piece_id=KNEK65KYMS5VO2REHA4UZ6WXFVKYT7LUQ47WPWDH6U6IWBYYFFXQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.427501339 (73h32m29.298584384s) [noise=true , address=185.167.204.206:4068 , piece_id=4ZBFLMHY2SZWQKT6SLGWH67EYB2UYOBVFW32X3NMEKJ2QGHUB62Q, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.431109811 (73h32m29.302192856s) [noise=true , address=185.167.204.205:4030 , piece_id=I5Y6S4GGMJK3HALRUWMSKQVLZMCSD5Q34GHLOJDKEOBWY7KLRMTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.432493316 (73h32m29.303576351s) [noise=true , address=185.167.204.206:4037 , piece_id=JWENQ4SQJCAD6RAEM35TQZNDXTMUOGV7JGWOLGA64BFQUF2MDKQQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.432695100 (73h32m29.303778135s) [noise=true , address=185.167.204.206:4070 , piece_id=RIILUOUAANPXHFPC67DDMBBXDRCIPMNGJKHPSGKBWPIKZLLB3SIQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.433119147 (73h32m29.304202192s) [noise=true , address=185.167.204.206:4039 , piece_id=56TVAHFZJLIWEWOWZCLQML2LX4IDFAERQQXRHDI2WAQPDDS36YNA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.434343859 (73h32m29.305426904s) [noise=true , address=185.167.204.206:4061 , piece_id=Y6BICZVIUK6OFLRXZJE3S4VVFIRWSAIJBAFOL6AEY66CCKZQFCTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.434887174 (73h32m29.305970209s) [noise=true , address=185.167.204.206:4064 , piece_id=NCWE447MXDE4SAN7MSMHMIU2TCDH2X7M5OJSICXKCE2Y3OFEACZA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.439186210 (73h32m29.310269255s) [noise=true , address=185.167.204.206:4050 , piece_id=BETXF3XMM5F42JB44YF2QGK6N54JOA6CZ4Z2EV5WOXZPUD2KKKTA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.439715649 (73h32m29.310798684s) [noise=true , address=185.167.204.206:4069 , piece_id=XGOYKYXSUAK4IHEM44RZB4DHNXYXIYQTVWZDNAPNE5G5HWUK4VFA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.440245988 (73h32m29.311329033s) [noise=true , address=185.167.204.205:4001 , piece_id=XAESPYQ55LBDJQ7WJIFWCVTCAGNXWAOOMSL3VMRERS3LZTMZB3GQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.440484813 (73h32m29.311567848s) [noise=true , address=185.167.204.206:4049 , piece_id=2NHQKCEPOXX475AS5SIZ7IXCNKJMXWW6LNU2OO6PBILRZ6DUZ5EA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.442183288 (73h32m29.313266323s) [noise=true , address=185.167.204.205:4015 , piece_id=FJKFMAGYH277I25J3WQ3IWR3DYRWICWQRC5LSKDDJ63XZ74S7MXQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.444441307 (73h32m29.315524352s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDPLG5LX3S7NJZDZSB4NQTMY4KBHBN3N5UGREBDB5JJHKBW3WWQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.445075635 (73h32m29.31615868s) [noise=true , address=185.167.204.205:4025 , piece_id=6VYO47IF26LZ5LO7KSUMRFMWOL6PKBOGT75UQQGCQUYX6CEFOVIQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.448313100 (73h32m29.319396135s) [noise=true , address=185.167.204.206:4048 , piece_id=BU4IJLUUZ66CLPARWCCMRZCCDQYHYWZOGK2EASU3HRNWLQ7B5RUA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.450008007 (73h32m29.321091042s) [noise=true , address=185.167.204.206:4057 , piece_id=ZLIWTLFQD4KDPMWOKRZUAVTK4BUF2J3BH6ETCMWBC7HMN5JBUQJA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.464765288 (73h32m29.335848323s) [noise=true , address=185.167.204.205:4007 , piece_id=45SASPPJTGKEBQTAINBPGKBUCTPJH4QXLGP34TK5H6MF5BK5AUAA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.006798349 (73h32m29.877881394s) [noise=true , address=81.171.14.70:4044 , piece_id=I4DPAS3D7CK5K5RMS7KASK775IF2QTC5SLDRNEXERI7CPHACCSMA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.052457282 (73h32m29.923540327s) [noise=true , address=81.171.14.70:4026 , piece_id=KL7AYBMWQ5P63JLIQPHVHBVZOFTOMSS2XJF2QTTIRQGF7ROTYPTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.087401313 (73h32m29.958484358s) [noise=true , address=81.171.14.70:4027 , piece_id=3XX2VTW3KPECPWINTSTFFUQL5CRKO4GENBS2TVTNSIN44YXCETJA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.150558347 (73h32m30.021641392s) [noise=true , address=81.171.14.70:4032 , piece_id=ZMDAJH2OX64GP4YR5E7CBNJC72RDTRMTCXEW67KIGQIZHJBO3ECQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.150583825 (73h32m30.02166687s) [seg_pos={0 0}, upload=64365]: Segment reached optimal threshold of 26 pieces. Dec 8 17:01:30.150889357 (73h32m30.021972412s) [noise=true , address=89.149.201.147:4016 , piece_id=BQT6TNPFUVT2OTT4YIRIC3CNAPIANWHIT23W5JPPLK6Y542ZER7A, seg_pos={0 0}, upload=64365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection; write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection Dec 8 17:01:30.150901891 (73h32m30.021984936s) [noise=true , address=89.149.201.147:4016 , piece_id=BQT6TNPFUVT2OTT4YIRIC3CNAPIANWHIT23W5JPPLK6Y542ZER7A, seg_pos={0 0}, upload=64365]: Long tail cancellation due to optimal threshold Dec 8 17:01:30.150914835 (73h32m30.02199788s) [noise=true , address=89.149.201.147:4011 , piece_id=BEU4ZFPKV5CFS3EU5FL5IOBSR2462IN5SUPKAEAWPGAHXWELR5AA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: read tcp 188.166.225.170:55962->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:01:30.150922571 (73h32m30.022005626s) [noise=true , address=89.149.201.147:4011 , piece_id=BEU4ZFPKV5CFS3EU5FL5IOBSR2462IN5SUPKAEAWPGAHXWELR5AA, seg_pos={0 0}, upload=64365]: Long tail cancellation due to optimal threshold Dec 8 17:01:30.150944682 (73h32m30.022027727s) [seg_pos={0 0}, upload=64365]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 17:01:30.151058490 (73h32m30.022141535s) [upload=64365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:01:30.151066825 (73h32m30.02214987s) [upload=64365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:03:21.289875989 (73h34m21.160959045s) [upload=64365]: Done uploading segments... Dec 8 17:03:21.289984466 (73h34m21.161067531s) [upload=64365]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.587753328s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.377008886s trace: https://jaeger.dev.storj.io/trace/136ff0f218b6bc4c FAIL (2025-12-08 17:43:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.377044895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.04107662s trace: https://jaeger.dev.storj.io/trace/46021ff75bd7decc FAIL (2025-12-08 17:43:57 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.041109302s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 4.857757049s trace: https://jaeger.dev.storj.io/trace/7058122c2f3fc553 FAIL (2025-12-08 17:43:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.857790391s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.689183212s trace: https://jaeger.dev.storj.io/trace/2bacc06da04f4505 FAIL (2025-12-08 17:43:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=185.167.204.206:4049 , piece_id=2NHQKCEPOXX475AS5SIZ7IXCNKJMXWW6LNU2OO6PBILRZ6DUZ5EA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.188203846 (73h32m28.059286901s) [noise=true , address=185.167.204.206:4069 , piece_id=XGOYKYXSUAK4IHEM44RZB4DHNXYXIYQTVWZDNAPNE5G5HWUK4VFA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.189671170 (73h32m28.060754215s) [noise=true , address=185.167.204.206:4050 , piece_id=BETXF3XMM5F42JB44YF2QGK6N54JOA6CZ4Z2EV5WOXZPUD2KKKTA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.191039326 (73h32m28.062122361s) [noise=true , address=81.171.14.70:4044 , piece_id=I4DPAS3D7CK5K5RMS7KASK775IF2QTC5SLDRNEXERI7CPHACCSMA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.193557270 (73h32m28.064640305s) [noise=true , address=185.167.204.206:4061 , piece_id=Y6BICZVIUK6OFLRXZJE3S4VVFIRWSAIJBAFOL6AEY66CCKZQFCTQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.195856628 (73h32m28.066939674s) [noise=true , address=185.167.204.205:4015 , piece_id=FJKFMAGYH277I25J3WQ3IWR3DYRWICWQRC5LSKDDJ63XZ74S7MXQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.198414219 (73h32m28.069497264s) [noise=true , address=185.167.204.205:4030 , piece_id=I5Y6S4GGMJK3HALRUWMSKQVLZMCSD5Q34GHLOJDKEOBWY7KLRMTQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.201095685 (73h32m28.07217873s) [noise=true , address=81.171.14.70:4032 , piece_id=ZMDAJH2OX64GP4YR5E7CBNJC72RDTRMTCXEW67KIGQIZHJBO3ECQ, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:28.204972177 (73h32m28.076055212s) [noise=true , address=185.167.204.206:4064 , piece_id=NCWE447MXDE4SAN7MSMHMIU2TCDH2X7M5OJSICXKCE2Y3OFEACZA, seg_pos={0 0}, upload=64365]: Uploading piece... Dec 8 17:01:29.014501224 (73h32m28.885584269s) [upload=64365]: Next returned nil segment Dec 8 17:01:29.014527022 (73h32m28.885610067s) [upload=64365]: Waiting for error group managing segments... Dec 8 17:01:29.408440684 (73h32m29.279523729s) [noise=true , address=185.167.204.205:4023 , piece_id=V344WVXRV2SEML6TEIASWYT4427PRE56R3KLFQJZ4ZNUFCQTNF7Q, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.409778682 (73h32m29.280861728s) [noise=true , address=185.167.204.205:4034 , piece_id=OGTA4AK7H6XFMPSB2HDYAYMVUHLMFU5IYM3TKOFWWDZBHYFAAESQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.411240316 (73h32m29.282323361s) [noise=true , address=185.167.204.206:4053 , piece_id=4MMKVKWPWU6CK5M7R74PXGQ34PPUXGWUJT2LJ6ZEXV6A2X72PJEQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.412578493 (73h32m29.283661538s) [noise=true , address=185.167.204.205:4036 , piece_id=6VYQ5BV7BKIGEIHKBERSBYRKTP7VC7M6NPNKWRSYSRVP243OVV4A, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.419000773 (73h32m29.290083818s) [noise=true , address=185.167.204.205:4029 , piece_id=KNEK65KYMS5VO2REHA4UZ6WXFVKYT7LUQ47WPWDH6U6IWBYYFFXQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.427501339 (73h32m29.298584384s) [noise=true , address=185.167.204.206:4068 , piece_id=4ZBFLMHY2SZWQKT6SLGWH67EYB2UYOBVFW32X3NMEKJ2QGHUB62Q, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.431109811 (73h32m29.302192856s) [noise=true , address=185.167.204.205:4030 , piece_id=I5Y6S4GGMJK3HALRUWMSKQVLZMCSD5Q34GHLOJDKEOBWY7KLRMTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.432493316 (73h32m29.303576351s) [noise=true , address=185.167.204.206:4037 , piece_id=JWENQ4SQJCAD6RAEM35TQZNDXTMUOGV7JGWOLGA64BFQUF2MDKQQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.432695100 (73h32m29.303778135s) [noise=true , address=185.167.204.206:4070 , piece_id=RIILUOUAANPXHFPC67DDMBBXDRCIPMNGJKHPSGKBWPIKZLLB3SIQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.433119147 (73h32m29.304202192s) [noise=true , address=185.167.204.206:4039 , piece_id=56TVAHFZJLIWEWOWZCLQML2LX4IDFAERQQXRHDI2WAQPDDS36YNA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.434343859 (73h32m29.305426904s) [noise=true , address=185.167.204.206:4061 , piece_id=Y6BICZVIUK6OFLRXZJE3S4VVFIRWSAIJBAFOL6AEY66CCKZQFCTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.434887174 (73h32m29.305970209s) [noise=true , address=185.167.204.206:4064 , piece_id=NCWE447MXDE4SAN7MSMHMIU2TCDH2X7M5OJSICXKCE2Y3OFEACZA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.439186210 (73h32m29.310269255s) [noise=true , address=185.167.204.206:4050 , piece_id=BETXF3XMM5F42JB44YF2QGK6N54JOA6CZ4Z2EV5WOXZPUD2KKKTA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.439715649 (73h32m29.310798684s) [noise=true , address=185.167.204.206:4069 , piece_id=XGOYKYXSUAK4IHEM44RZB4DHNXYXIYQTVWZDNAPNE5G5HWUK4VFA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.440245988 (73h32m29.311329033s) [noise=true , address=185.167.204.205:4001 , piece_id=XAESPYQ55LBDJQ7WJIFWCVTCAGNXWAOOMSL3VMRERS3LZTMZB3GQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.440484813 (73h32m29.311567848s) [noise=true , address=185.167.204.206:4049 , piece_id=2NHQKCEPOXX475AS5SIZ7IXCNKJMXWW6LNU2OO6PBILRZ6DUZ5EA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.442183288 (73h32m29.313266323s) [noise=true , address=185.167.204.205:4015 , piece_id=FJKFMAGYH277I25J3WQ3IWR3DYRWICWQRC5LSKDDJ63XZ74S7MXQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.444441307 (73h32m29.315524352s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDPLG5LX3S7NJZDZSB4NQTMY4KBHBN3N5UGREBDB5JJHKBW3WWQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.445075635 (73h32m29.31615868s) [noise=true , address=185.167.204.205:4025 , piece_id=6VYO47IF26LZ5LO7KSUMRFMWOL6PKBOGT75UQQGCQUYX6CEFOVIQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.448313100 (73h32m29.319396135s) [noise=true , address=185.167.204.206:4048 , piece_id=BU4IJLUUZ66CLPARWCCMRZCCDQYHYWZOGK2EASU3HRNWLQ7B5RUA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.450008007 (73h32m29.321091042s) [noise=true , address=185.167.204.206:4057 , piece_id=ZLIWTLFQD4KDPMWOKRZUAVTK4BUF2J3BH6ETCMWBC7HMN5JBUQJA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:29.464765288 (73h32m29.335848323s) [noise=true , address=185.167.204.205:4007 , piece_id=45SASPPJTGKEBQTAINBPGKBUCTPJH4QXLGP34TK5H6MF5BK5AUAA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.006798349 (73h32m29.877881394s) [noise=true , address=81.171.14.70:4044 , piece_id=I4DPAS3D7CK5K5RMS7KASK775IF2QTC5SLDRNEXERI7CPHACCSMA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.052457282 (73h32m29.923540327s) [noise=true , address=81.171.14.70:4026 , piece_id=KL7AYBMWQ5P63JLIQPHVHBVZOFTOMSS2XJF2QTTIRQGF7ROTYPTQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.087401313 (73h32m29.958484358s) [noise=true , address=81.171.14.70:4027 , piece_id=3XX2VTW3KPECPWINTSTFFUQL5CRKO4GENBS2TVTNSIN44YXCETJA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.150558347 (73h32m30.021641392s) [noise=true , address=81.171.14.70:4032 , piece_id=ZMDAJH2OX64GP4YR5E7CBNJC72RDTRMTCXEW67KIGQIZHJBO3ECQ, seg_pos={0 0}, upload=64365]: Done uploading piece. err: <nil> Dec 8 17:01:30.150583825 (73h32m30.02166687s) [seg_pos={0 0}, upload=64365]: Segment reached optimal threshold of 26 pieces. Dec 8 17:01:30.150889357 (73h32m30.021972412s) [noise=true , address=89.149.201.147:4016 , piece_id=BQT6TNPFUVT2OTT4YIRIC3CNAPIANWHIT23W5JPPLK6Y542ZER7A, seg_pos={0 0}, upload=64365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection; write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33966->89.149.201.147:4016: use of closed network connection Dec 8 17:01:30.150901891 (73h32m30.021984936s) [noise=true , address=89.149.201.147:4016 , piece_id=BQT6TNPFUVT2OTT4YIRIC3CNAPIANWHIT23W5JPPLK6Y542ZER7A, seg_pos={0 0}, upload=64365]: Long tail cancellation due to optimal threshold Dec 8 17:01:30.150914835 (73h32m30.02199788s) [noise=true , address=89.149.201.147:4011 , piece_id=BEU4ZFPKV5CFS3EU5FL5IOBSR2462IN5SUPKAEAWPGAHXWELR5AA, seg_pos={0 0}, upload=64365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: read tcp 188.166.225.170:55962->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:01:30.150922571 (73h32m30.022005626s) [noise=true , address=89.149.201.147:4011 , piece_id=BEU4ZFPKV5CFS3EU5FL5IOBSR2462IN5SUPKAEAWPGAHXWELR5AA, seg_pos={0 0}, upload=64365]: Long tail cancellation due to optimal threshold Dec 8 17:01:30.150944682 (73h32m30.022027727s) [seg_pos={0 0}, upload=64365]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 17:01:30.151058490 (73h32m30.022141535s) [upload=64365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:01:30.151066825 (73h32m30.02214987s) [upload=64365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:03:21.289875989 (73h34m21.160959045s) [upload=64365]: Done uploading segments... Dec 8 17:03:21.289984466 (73h34m21.161067531s) [upload=64365]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.689207767s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.032622673s trace: https://jaeger.dev.storj.io/trace/71acc8d1f58c5156 FAIL (2025-12-08 17:43:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:19:35.057390695 (73h50m34.92847375s) [upload=64625]: Starting upload Dec 8 17:19:35.057451541 (73h50m34.928534586s) [upload=64625]: Done starting upload Dec 8 17:19:35.057525381 (73h50m34.928608426s) [upload=64625]: Uploading segments... Dec 8 17:19:35.057592810 (73h50m34.928675855s) [upload=64625]: Got next segment. Inline: true Dec 8 17:19:35.057598100 (73h50m34.928681145s) [upload=64625]: Waiting for error group managing segments... Dec 8 17:19:35.057643065 (73h50m34.92872611s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:19:35.057648215 (73h50m34.92873126s) [upload=64625]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 17:19:35.057651671 (73h50m34.928734716s) [upload=64625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:19:35.334831026 (73h50m35.205914061s) [upload=64625]: Done uploading segments... Dec 8 17:19:35.334853018 (73h50m35.205936063s) [upload=64625]: Upload finished. err: <nil> Duration:* 2.03265801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.428358926s trace: https://jaeger.dev.storj.io/trace/136ff0f218b6bc4c FAIL (2025-12-08 17:43:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428384294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.821686072s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 17:43:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.821713495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.........X......................X.........] duration: 5.905354023s trace: https://jaeger.dev.storj.io/trace/7f2371936424c2ff FAIL (2025-12-08 17:43:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.905383149s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.850107623s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 17:43:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.850133391s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)