Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.34024861s trace: https://jaeger.dev.storj.io/trace/2bd986c05c15b190 FAIL (2025-12-08 18:10:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== CDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.842730397 (74h24m40.713813432s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.843290634 (74h24m40.714373669s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.845386525 (74h24m40.716469571s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.846892383 (74h24m40.717975428s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.848772573 (74h24m40.719855618s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.850191745 (74h24m40.72127478s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.851609155 (74h24m40.7226922s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.852832574 (74h24m40.723915619s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.854147148 (74h24m40.725230203s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:41.670064128 (74h24m41.541147173s) [upload=65114]: Next returned nil segment Dec 8 17:53:41.670145664 (74h24m41.541228709s) [upload=65114]: Waiting for error group managing segments... Dec 8 17:53:42.057773064 (74h24m41.928856099s) [noise=true , address=185.167.204.206:4065 , piece_id=7GQAOZEZ6OJKW3UK7U3NXYZM7VXGPHKW6UKHSTEUKJDNRZAYUA4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058200909 (74h24m41.929283954s) [noise=true , address=185.167.204.206:4045 , piece_id=6O2I5IIAPPQFG66OPIOVY2GOHO4C5CM2UXMJHGY45WGQVZK3JGPQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058523583 (74h24m41.929606628s) [noise=true , address=185.167.204.205:4034 , piece_id=6JYH3N55MJSMXE6VNGSZK7DZX7KUV7CPELNHIMPSFOXG4U2NXRTQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.060741817 (74h24m41.931824863s) [noise=true , address=185.167.204.205:4025 , piece_id=2MDDUA2MUJ6MXICI5H5O4KPAIRJUL7QLBIC4L6MIEKNDYQIO2EJQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.062816829 (74h24m41.933899864s) [noise=true , address=185.167.204.206:4072 , piece_id=X5XLPXKRFRIEQ32PZKY23AS4CFBXSC5NVCJF24IY2LQH4KLGG6RA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.064906518 (74h24m41.935989563s) [noise=true , address=185.167.204.206:4059 , piece_id=KPZWLMWMZA7GCT6JWIMSMLGUW6UVJ3FPWJKMSHP72C42TH5Z4ITA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.065603074 (74h24m41.93668611s) [noise=true , address=185.167.204.206:4047 , piece_id=4G5HDKWP26ER64KVLUXJ2IMQJUXMQ4YLNBYQAMY7MQMGE2PSREEA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.066790916 (74h24m41.937873951s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.075608666 (74h24m41.946691701s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.076342644 (74h24m41.947425679s) [noise=true , address=185.167.204.205:4036 , piece_id=FKH55HUCJTNREM6PBSFGWURF5HIGWAM3U6OPQQVTRGEUA6K7JM2Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.077612843 (74h24m41.948695888s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.078994745 (74h24m41.95007778s) [noise=true , address=185.167.204.206:4046 , piece_id=SDP7J6VHYTCJSKOHR4AXONUYML7VNYQIVATGTPENHWS4F4CO3A4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.082062506 (74h24m41.953145541s) [noise=true , address=185.167.204.206:4070 , piece_id=5ETK46YMRH46TMYDEYR4CPV3XDF3FS5MR5RSOYPZKD33EXMS5TQQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.096426879 (74h24m41.967509924s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.766933610 (74h24m42.638016655s) [noise=true , address=81.171.14.70:4031 , piece_id=S4T3ORC533X2OW2Y3BQDAFSYXBTD3THLREQMMJ2G7QPM3UDGHRCQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.860649744 (74h24m42.731732789s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.874184617 (74h24m42.745267652s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.254322126 (74h24m44.125405172s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.671496219 (74h24m44.542579254s) [noise=true , address=81.171.14.70:4025 , piece_id=YX5EYUXYW4ZY3L6CH673H5BSHKUEKTUGGKVSFVENUVJBF3ONZUKA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.750460513 (74h24m44.621543558s) [noise=true , address=81.171.14.70:4033 , piece_id=LSIJ6NCTBALZH2QRPFFTLOXKKWF7J2E3NL2GIYXCDF2RC5KMNBPA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.792368875 (74h24m44.66345192s) [noise=true , address=81.171.14.70:4027 , piece_id=X53DOOCXSAHI64TN5GNSEOHBJK7GL6TLL6W7NJM5KQIVW3ITZ2BA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:45.851581290 (74h24m45.722664326s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.587984219 (74h24m46.459067264s) [noise=true , address=89.149.201.147:4022 , piece_id=ZEX4GP43WCDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.808512099 (74h24m46.679595144s) [noise=true , address=81.171.14.70:4037 , piece_id=MHNTZCXG6COLI7J45SSYK6SZDVSSEBXOFFY7KEGP7NJUDEVLX7NA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:47.152614810 (74h24m47.023697845s) [noise=true , address=81.171.14.70:4038 , piece_id=VFW2FCTINXQNF3AFAFQW5AYWC2EZ2P52ERVUFU5DMH7672AWAQTA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686376464 (74h24m49.557459499s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686397555 (74h24m49.55748059s) [seg_pos={0 0}, upload=65114]: Segment reached optimal threshold of 26 pieces. Dec 8 17:53:49.686927403 (74h24m49.558010438s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: read tcp 188.166.225.170:60652->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687035098 (74h24m49.558118143s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687151320 (74h24m49.558234365s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8): protocol: read tcp 188.166.225.170:58886->89.149.201.147:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687225261 (74h24m49.558308326s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687301826 (74h24m49.558384872s) [seg_pos={0 0}, upload=65114]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 17:53:49.687437044 (74h24m49.558520089s) [upload=65114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:53:49.687467202 (74h24m49.558550247s) [upload=65114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 18:01:19.776524628 (74h32m19.647607673s) [upload=65114]: Done uploading segments... Dec 8 18:01:19.776620922 (74h32m19.647703967s) [upload=65114]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.34028609s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X..X...............] duration: 3.527292186s trace: https://jaeger.dev.storj.io/trace/2a639037b48061f3 FAIL (2025-12-08 18:10:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527325198s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......................X.......] duration: 5.745992841s trace: https://jaeger.dev.storj.io/trace/ab29194e90e610a FAIL (2025-12-08 18:09:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.746018311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.946656732s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 18:09:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.946686327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.....................X........X........] duration: 5.098855222s trace: https://jaeger.dev.storj.io/trace/355abf85f96191de FAIL (2025-12-08 18:09:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.098883085s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X......................X......] duration: 5.999896711s trace: https://jaeger.dev.storj.io/trace/ab29194e90e610a FAIL (2025-12-08 18:09:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.999924783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.169989007s trace: https://jaeger.dev.storj.io/trace/54ea49bbbba7a774 FAIL (2025-12-08 18:09:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.170020166s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.917102385s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 18:09:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.917126952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X............X...........................] duration: 6.095029057s trace: https://jaeger.dev.storj.io/trace/9e19cf33ba5771 FAIL (2025-12-08 18:09:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.095055598s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.871406802s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 18:08:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.871436098s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.282717642s trace: https://jaeger.dev.storj.io/trace/2bd986c05c15b190 FAIL (2025-12-08 18:08:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== CDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.842730397 (74h24m40.713813432s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.843290634 (74h24m40.714373669s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.845386525 (74h24m40.716469571s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.846892383 (74h24m40.717975428s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.848772573 (74h24m40.719855618s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.850191745 (74h24m40.72127478s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.851609155 (74h24m40.7226922s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.852832574 (74h24m40.723915619s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.854147148 (74h24m40.725230203s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:41.670064128 (74h24m41.541147173s) [upload=65114]: Next returned nil segment Dec 8 17:53:41.670145664 (74h24m41.541228709s) [upload=65114]: Waiting for error group managing segments... Dec 8 17:53:42.057773064 (74h24m41.928856099s) [noise=true , address=185.167.204.206:4065 , piece_id=7GQAOZEZ6OJKW3UK7U3NXYZM7VXGPHKW6UKHSTEUKJDNRZAYUA4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058200909 (74h24m41.929283954s) [noise=true , address=185.167.204.206:4045 , piece_id=6O2I5IIAPPQFG66OPIOVY2GOHO4C5CM2UXMJHGY45WGQVZK3JGPQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058523583 (74h24m41.929606628s) [noise=true , address=185.167.204.205:4034 , piece_id=6JYH3N55MJSMXE6VNGSZK7DZX7KUV7CPELNHIMPSFOXG4U2NXRTQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.060741817 (74h24m41.931824863s) [noise=true , address=185.167.204.205:4025 , piece_id=2MDDUA2MUJ6MXICI5H5O4KPAIRJUL7QLBIC4L6MIEKNDYQIO2EJQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.062816829 (74h24m41.933899864s) [noise=true , address=185.167.204.206:4072 , piece_id=X5XLPXKRFRIEQ32PZKY23AS4CFBXSC5NVCJF24IY2LQH4KLGG6RA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.064906518 (74h24m41.935989563s) [noise=true , address=185.167.204.206:4059 , piece_id=KPZWLMWMZA7GCT6JWIMSMLGUW6UVJ3FPWJKMSHP72C42TH5Z4ITA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.065603074 (74h24m41.93668611s) [noise=true , address=185.167.204.206:4047 , piece_id=4G5HDKWP26ER64KVLUXJ2IMQJUXMQ4YLNBYQAMY7MQMGE2PSREEA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.066790916 (74h24m41.937873951s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.075608666 (74h24m41.946691701s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.076342644 (74h24m41.947425679s) [noise=true , address=185.167.204.205:4036 , piece_id=FKH55HUCJTNREM6PBSFGWURF5HIGWAM3U6OPQQVTRGEUA6K7JM2Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.077612843 (74h24m41.948695888s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.078994745 (74h24m41.95007778s) [noise=true , address=185.167.204.206:4046 , piece_id=SDP7J6VHYTCJSKOHR4AXONUYML7VNYQIVATGTPENHWS4F4CO3A4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.082062506 (74h24m41.953145541s) [noise=true , address=185.167.204.206:4070 , piece_id=5ETK46YMRH46TMYDEYR4CPV3XDF3FS5MR5RSOYPZKD33EXMS5TQQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.096426879 (74h24m41.967509924s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.766933610 (74h24m42.638016655s) [noise=true , address=81.171.14.70:4031 , piece_id=S4T3ORC533X2OW2Y3BQDAFSYXBTD3THLREQMMJ2G7QPM3UDGHRCQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.860649744 (74h24m42.731732789s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.874184617 (74h24m42.745267652s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.254322126 (74h24m44.125405172s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.671496219 (74h24m44.542579254s) [noise=true , address=81.171.14.70:4025 , piece_id=YX5EYUXYW4ZY3L6CH673H5BSHKUEKTUGGKVSFVENUVJBF3ONZUKA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.750460513 (74h24m44.621543558s) [noise=true , address=81.171.14.70:4033 , piece_id=LSIJ6NCTBALZH2QRPFFTLOXKKWF7J2E3NL2GIYXCDF2RC5KMNBPA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.792368875 (74h24m44.66345192s) [noise=true , address=81.171.14.70:4027 , piece_id=X53DOOCXSAHI64TN5GNSEOHBJK7GL6TLL6W7NJM5KQIVW3ITZ2BA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:45.851581290 (74h24m45.722664326s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.587984219 (74h24m46.459067264s) [noise=true , address=89.149.201.147:4022 , piece_id=ZEX4GP43WCDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.808512099 (74h24m46.679595144s) [noise=true , address=81.171.14.70:4037 , piece_id=MHNTZCXG6COLI7J45SSYK6SZDVSSEBXOFFY7KEGP7NJUDEVLX7NA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:47.152614810 (74h24m47.023697845s) [noise=true , address=81.171.14.70:4038 , piece_id=VFW2FCTINXQNF3AFAFQW5AYWC2EZ2P52ERVUFU5DMH7672AWAQTA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686376464 (74h24m49.557459499s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686397555 (74h24m49.55748059s) [seg_pos={0 0}, upload=65114]: Segment reached optimal threshold of 26 pieces. Dec 8 17:53:49.686927403 (74h24m49.558010438s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: read tcp 188.166.225.170:60652->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687035098 (74h24m49.558118143s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687151320 (74h24m49.558234365s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8): protocol: read tcp 188.166.225.170:58886->89.149.201.147:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687225261 (74h24m49.558308326s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687301826 (74h24m49.558384872s) [seg_pos={0 0}, upload=65114]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 17:53:49.687437044 (74h24m49.558520089s) [upload=65114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:53:49.687467202 (74h24m49.558550247s) [upload=65114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 18:01:19.776524628 (74h32m19.647607673s) [upload=65114]: Done uploading segments... Dec 8 18:01:19.776620922 (74h32m19.647703967s) [upload=65114]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.282744946s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X.........................................X..X..............] duration: 3.4235815s trace: https://jaeger.dev.storj.io/trace/2a639037b48061f3 FAIL (2025-12-08 18:08:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.423723179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.....................X........X.......] duration: 5.273508989s trace: https://jaeger.dev.storj.io/trace/355abf85f96191de FAIL (2025-12-08 18:08:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.273542413s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.667820014s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 18:08:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.66784467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X............X..........................] duration: 5.885699178s trace: https://jaeger.dev.storj.io/trace/9e19cf33ba5771 FAIL (2025-12-08 18:08:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.885729306s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 34.582218032s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 18:08:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 34.582242908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X.........................................X..X.............] duration: 4.317054877s trace: https://jaeger.dev.storj.io/trace/2a639037b48061f3 FAIL (2025-12-08 18:08:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.317089613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.038465751s trace: https://jaeger.dev.storj.io/trace/54ea49bbbba7a774 FAIL (2025-12-08 18:08:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.038584828s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X......................X.....] duration: 5.75492146s trace: https://jaeger.dev.storj.io/trace/ab29194e90e610a FAIL (2025-12-08 18:08:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.754948913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.824354144s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 18:08:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.824380444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.016854912s trace: https://jaeger.dev.storj.io/trace/54ea49bbbba7a774 FAIL (2025-12-08 18:07:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.016881792s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.660183588s trace: https://jaeger.dev.storj.io/trace/2bd986c05c15b190 FAIL (2025-12-08 18:07:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== CDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.842730397 (74h24m40.713813432s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.843290634 (74h24m40.714373669s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.845386525 (74h24m40.716469571s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.846892383 (74h24m40.717975428s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.848772573 (74h24m40.719855618s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.850191745 (74h24m40.72127478s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.851609155 (74h24m40.7226922s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.852832574 (74h24m40.723915619s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.854147148 (74h24m40.725230203s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:41.670064128 (74h24m41.541147173s) [upload=65114]: Next returned nil segment Dec 8 17:53:41.670145664 (74h24m41.541228709s) [upload=65114]: Waiting for error group managing segments... Dec 8 17:53:42.057773064 (74h24m41.928856099s) [noise=true , address=185.167.204.206:4065 , piece_id=7GQAOZEZ6OJKW3UK7U3NXYZM7VXGPHKW6UKHSTEUKJDNRZAYUA4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058200909 (74h24m41.929283954s) [noise=true , address=185.167.204.206:4045 , piece_id=6O2I5IIAPPQFG66OPIOVY2GOHO4C5CM2UXMJHGY45WGQVZK3JGPQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058523583 (74h24m41.929606628s) [noise=true , address=185.167.204.205:4034 , piece_id=6JYH3N55MJSMXE6VNGSZK7DZX7KUV7CPELNHIMPSFOXG4U2NXRTQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.060741817 (74h24m41.931824863s) [noise=true , address=185.167.204.205:4025 , piece_id=2MDDUA2MUJ6MXICI5H5O4KPAIRJUL7QLBIC4L6MIEKNDYQIO2EJQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.062816829 (74h24m41.933899864s) [noise=true , address=185.167.204.206:4072 , piece_id=X5XLPXKRFRIEQ32PZKY23AS4CFBXSC5NVCJF24IY2LQH4KLGG6RA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.064906518 (74h24m41.935989563s) [noise=true , address=185.167.204.206:4059 , piece_id=KPZWLMWMZA7GCT6JWIMSMLGUW6UVJ3FPWJKMSHP72C42TH5Z4ITA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.065603074 (74h24m41.93668611s) [noise=true , address=185.167.204.206:4047 , piece_id=4G5HDKWP26ER64KVLUXJ2IMQJUXMQ4YLNBYQAMY7MQMGE2PSREEA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.066790916 (74h24m41.937873951s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.075608666 (74h24m41.946691701s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.076342644 (74h24m41.947425679s) [noise=true , address=185.167.204.205:4036 , piece_id=FKH55HUCJTNREM6PBSFGWURF5HIGWAM3U6OPQQVTRGEUA6K7JM2Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.077612843 (74h24m41.948695888s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.078994745 (74h24m41.95007778s) [noise=true , address=185.167.204.206:4046 , piece_id=SDP7J6VHYTCJSKOHR4AXONUYML7VNYQIVATGTPENHWS4F4CO3A4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.082062506 (74h24m41.953145541s) [noise=true , address=185.167.204.206:4070 , piece_id=5ETK46YMRH46TMYDEYR4CPV3XDF3FS5MR5RSOYPZKD33EXMS5TQQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.096426879 (74h24m41.967509924s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.766933610 (74h24m42.638016655s) [noise=true , address=81.171.14.70:4031 , piece_id=S4T3ORC533X2OW2Y3BQDAFSYXBTD3THLREQMMJ2G7QPM3UDGHRCQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.860649744 (74h24m42.731732789s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.874184617 (74h24m42.745267652s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.254322126 (74h24m44.125405172s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.671496219 (74h24m44.542579254s) [noise=true , address=81.171.14.70:4025 , piece_id=YX5EYUXYW4ZY3L6CH673H5BSHKUEKTUGGKVSFVENUVJBF3ONZUKA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.750460513 (74h24m44.621543558s) [noise=true , address=81.171.14.70:4033 , piece_id=LSIJ6NCTBALZH2QRPFFTLOXKKWF7J2E3NL2GIYXCDF2RC5KMNBPA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.792368875 (74h24m44.66345192s) [noise=true , address=81.171.14.70:4027 , piece_id=X53DOOCXSAHI64TN5GNSEOHBJK7GL6TLL6W7NJM5KQIVW3ITZ2BA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:45.851581290 (74h24m45.722664326s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.587984219 (74h24m46.459067264s) [noise=true , address=89.149.201.147:4022 , piece_id=ZEX4GP43WCDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.808512099 (74h24m46.679595144s) [noise=true , address=81.171.14.70:4037 , piece_id=MHNTZCXG6COLI7J45SSYK6SZDVSSEBXOFFY7KEGP7NJUDEVLX7NA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:47.152614810 (74h24m47.023697845s) [noise=true , address=81.171.14.70:4038 , piece_id=VFW2FCTINXQNF3AFAFQW5AYWC2EZ2P52ERVUFU5DMH7672AWAQTA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686376464 (74h24m49.557459499s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686397555 (74h24m49.55748059s) [seg_pos={0 0}, upload=65114]: Segment reached optimal threshold of 26 pieces. Dec 8 17:53:49.686927403 (74h24m49.558010438s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: read tcp 188.166.225.170:60652->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687035098 (74h24m49.558118143s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687151320 (74h24m49.558234365s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8): protocol: read tcp 188.166.225.170:58886->89.149.201.147:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687225261 (74h24m49.558308326s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687301826 (74h24m49.558384872s) [seg_pos={0 0}, upload=65114]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 17:53:49.687437044 (74h24m49.558520089s) [upload=65114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:53:49.687467202 (74h24m49.558550247s) [upload=65114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 18:01:19.776524628 (74h32m19.647607673s) [upload=65114]: Done uploading segments... Dec 8 18:01:19.776620922 (74h32m19.647703967s) [upload=65114]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.660213525s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.....................X........X......] duration: 5.128671683s trace: https://jaeger.dev.storj.io/trace/355abf85f96191de FAIL (2025-12-08 18:07:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.128701672s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X.........................................X..X............] duration: 3.4624381s trace: https://jaeger.dev.storj.io/trace/2a639037b48061f3 FAIL (2025-12-08 18:07:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462472867s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X......................X....] duration: 5.554998489s trace: https://jaeger.dev.storj.io/trace/ab29194e90e610a FAIL (2025-12-08 18:07:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.555019258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X............X.........................] duration: 6.328627981s trace: https://jaeger.dev.storj.io/trace/9e19cf33ba5771 FAIL (2025-12-08 18:07:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.328656605s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.983082374s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 18:07:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.983101049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.010411693s trace: https://jaeger.dev.storj.io/trace/54ea49bbbba7a774 FAIL (2025-12-08 18:07:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.010441359s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.815344929s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 18:07:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.815368385s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X......................X...] duration: 6.118916867s trace: https://jaeger.dev.storj.io/trace/ab29194e90e610a FAIL (2025-12-08 18:07:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.118951152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.....................X........X.....] duration: 5.250574064s trace: https://jaeger.dev.storj.io/trace/355abf85f96191de FAIL (2025-12-08 18:07:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.250592259s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.125318486s trace: https://jaeger.dev.storj.io/trace/2bd986c05c15b190 FAIL (2025-12-08 18:06:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== CDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.842730397 (74h24m40.713813432s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.843290634 (74h24m40.714373669s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.845386525 (74h24m40.716469571s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.846892383 (74h24m40.717975428s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.848772573 (74h24m40.719855618s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.850191745 (74h24m40.72127478s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.851609155 (74h24m40.7226922s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.852832574 (74h24m40.723915619s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.854147148 (74h24m40.725230203s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:41.670064128 (74h24m41.541147173s) [upload=65114]: Next returned nil segment Dec 8 17:53:41.670145664 (74h24m41.541228709s) [upload=65114]: Waiting for error group managing segments... Dec 8 17:53:42.057773064 (74h24m41.928856099s) [noise=true , address=185.167.204.206:4065 , piece_id=7GQAOZEZ6OJKW3UK7U3NXYZM7VXGPHKW6UKHSTEUKJDNRZAYUA4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058200909 (74h24m41.929283954s) [noise=true , address=185.167.204.206:4045 , piece_id=6O2I5IIAPPQFG66OPIOVY2GOHO4C5CM2UXMJHGY45WGQVZK3JGPQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058523583 (74h24m41.929606628s) [noise=true , address=185.167.204.205:4034 , piece_id=6JYH3N55MJSMXE6VNGSZK7DZX7KUV7CPELNHIMPSFOXG4U2NXRTQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.060741817 (74h24m41.931824863s) [noise=true , address=185.167.204.205:4025 , piece_id=2MDDUA2MUJ6MXICI5H5O4KPAIRJUL7QLBIC4L6MIEKNDYQIO2EJQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.062816829 (74h24m41.933899864s) [noise=true , address=185.167.204.206:4072 , piece_id=X5XLPXKRFRIEQ32PZKY23AS4CFBXSC5NVCJF24IY2LQH4KLGG6RA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.064906518 (74h24m41.935989563s) [noise=true , address=185.167.204.206:4059 , piece_id=KPZWLMWMZA7GCT6JWIMSMLGUW6UVJ3FPWJKMSHP72C42TH5Z4ITA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.065603074 (74h24m41.93668611s) [noise=true , address=185.167.204.206:4047 , piece_id=4G5HDKWP26ER64KVLUXJ2IMQJUXMQ4YLNBYQAMY7MQMGE2PSREEA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.066790916 (74h24m41.937873951s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.075608666 (74h24m41.946691701s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.076342644 (74h24m41.947425679s) [noise=true , address=185.167.204.205:4036 , piece_id=FKH55HUCJTNREM6PBSFGWURF5HIGWAM3U6OPQQVTRGEUA6K7JM2Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.077612843 (74h24m41.948695888s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.078994745 (74h24m41.95007778s) [noise=true , address=185.167.204.206:4046 , piece_id=SDP7J6VHYTCJSKOHR4AXONUYML7VNYQIVATGTPENHWS4F4CO3A4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.082062506 (74h24m41.953145541s) [noise=true , address=185.167.204.206:4070 , piece_id=5ETK46YMRH46TMYDEYR4CPV3XDF3FS5MR5RSOYPZKD33EXMS5TQQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.096426879 (74h24m41.967509924s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.766933610 (74h24m42.638016655s) [noise=true , address=81.171.14.70:4031 , piece_id=S4T3ORC533X2OW2Y3BQDAFSYXBTD3THLREQMMJ2G7QPM3UDGHRCQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.860649744 (74h24m42.731732789s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.874184617 (74h24m42.745267652s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.254322126 (74h24m44.125405172s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.671496219 (74h24m44.542579254s) [noise=true , address=81.171.14.70:4025 , piece_id=YX5EYUXYW4ZY3L6CH673H5BSHKUEKTUGGKVSFVENUVJBF3ONZUKA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.750460513 (74h24m44.621543558s) [noise=true , address=81.171.14.70:4033 , piece_id=LSIJ6NCTBALZH2QRPFFTLOXKKWF7J2E3NL2GIYXCDF2RC5KMNBPA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.792368875 (74h24m44.66345192s) [noise=true , address=81.171.14.70:4027 , piece_id=X53DOOCXSAHI64TN5GNSEOHBJK7GL6TLL6W7NJM5KQIVW3ITZ2BA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:45.851581290 (74h24m45.722664326s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.587984219 (74h24m46.459067264s) [noise=true , address=89.149.201.147:4022 , piece_id=ZEX4GP43WCDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.808512099 (74h24m46.679595144s) [noise=true , address=81.171.14.70:4037 , piece_id=MHNTZCXG6COLI7J45SSYK6SZDVSSEBXOFFY7KEGP7NJUDEVLX7NA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:47.152614810 (74h24m47.023697845s) [noise=true , address=81.171.14.70:4038 , piece_id=VFW2FCTINXQNF3AFAFQW5AYWC2EZ2P52ERVUFU5DMH7672AWAQTA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686376464 (74h24m49.557459499s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686397555 (74h24m49.55748059s) [seg_pos={0 0}, upload=65114]: Segment reached optimal threshold of 26 pieces. Dec 8 17:53:49.686927403 (74h24m49.558010438s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: read tcp 188.166.225.170:60652->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687035098 (74h24m49.558118143s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687151320 (74h24m49.558234365s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8): protocol: read tcp 188.166.225.170:58886->89.149.201.147:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687225261 (74h24m49.558308326s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687301826 (74h24m49.558384872s) [seg_pos={0 0}, upload=65114]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 17:53:49.687437044 (74h24m49.558520089s) [upload=65114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:53:49.687467202 (74h24m49.558550247s) [upload=65114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 18:01:19.776524628 (74h32m19.647607673s) [upload=65114]: Done uploading segments... Dec 8 18:01:19.776620922 (74h32m19.647703967s) [upload=65114]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.125346178s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.996704069s trace: https://jaeger.dev.storj.io/trace/54ea49bbbba7a774 FAIL (2025-12-08 18:06:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.996720291s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.687120294s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 18:06:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.687148307s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X.........................................X..X...........] duration: 3.379034294s trace: https://jaeger.dev.storj.io/trace/2a639037b48061f3 FAIL (2025-12-08 18:06:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.379080492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X............X........................] duration: 5.859352147s trace: https://jaeger.dev.storj.io/trace/9e19cf33ba5771 FAIL (2025-12-08 18:06:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.859414575s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.948550949s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 18:06:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.948575717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 18.539038535s trace: https://jaeger.dev.storj.io/trace/2bd986c05c15b190 FAIL (2025-12-08 18:06:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== CDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.842730397 (74h24m40.713813432s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.843290634 (74h24m40.714373669s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.845386525 (74h24m40.716469571s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.846892383 (74h24m40.717975428s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.848772573 (74h24m40.719855618s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.850191745 (74h24m40.72127478s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.851609155 (74h24m40.7226922s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.852832574 (74h24m40.723915619s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.854147148 (74h24m40.725230203s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:41.670064128 (74h24m41.541147173s) [upload=65114]: Next returned nil segment Dec 8 17:53:41.670145664 (74h24m41.541228709s) [upload=65114]: Waiting for error group managing segments... Dec 8 17:53:42.057773064 (74h24m41.928856099s) [noise=true , address=185.167.204.206:4065 , piece_id=7GQAOZEZ6OJKW3UK7U3NXYZM7VXGPHKW6UKHSTEUKJDNRZAYUA4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058200909 (74h24m41.929283954s) [noise=true , address=185.167.204.206:4045 , piece_id=6O2I5IIAPPQFG66OPIOVY2GOHO4C5CM2UXMJHGY45WGQVZK3JGPQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058523583 (74h24m41.929606628s) [noise=true , address=185.167.204.205:4034 , piece_id=6JYH3N55MJSMXE6VNGSZK7DZX7KUV7CPELNHIMPSFOXG4U2NXRTQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.060741817 (74h24m41.931824863s) [noise=true , address=185.167.204.205:4025 , piece_id=2MDDUA2MUJ6MXICI5H5O4KPAIRJUL7QLBIC4L6MIEKNDYQIO2EJQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.062816829 (74h24m41.933899864s) [noise=true , address=185.167.204.206:4072 , piece_id=X5XLPXKRFRIEQ32PZKY23AS4CFBXSC5NVCJF24IY2LQH4KLGG6RA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.064906518 (74h24m41.935989563s) [noise=true , address=185.167.204.206:4059 , piece_id=KPZWLMWMZA7GCT6JWIMSMLGUW6UVJ3FPWJKMSHP72C42TH5Z4ITA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.065603074 (74h24m41.93668611s) [noise=true , address=185.167.204.206:4047 , piece_id=4G5HDKWP26ER64KVLUXJ2IMQJUXMQ4YLNBYQAMY7MQMGE2PSREEA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.066790916 (74h24m41.937873951s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.075608666 (74h24m41.946691701s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.076342644 (74h24m41.947425679s) [noise=true , address=185.167.204.205:4036 , piece_id=FKH55HUCJTNREM6PBSFGWURF5HIGWAM3U6OPQQVTRGEUA6K7JM2Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.077612843 (74h24m41.948695888s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.078994745 (74h24m41.95007778s) [noise=true , address=185.167.204.206:4046 , piece_id=SDP7J6VHYTCJSKOHR4AXONUYML7VNYQIVATGTPENHWS4F4CO3A4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.082062506 (74h24m41.953145541s) [noise=true , address=185.167.204.206:4070 , piece_id=5ETK46YMRH46TMYDEYR4CPV3XDF3FS5MR5RSOYPZKD33EXMS5TQQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.096426879 (74h24m41.967509924s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.766933610 (74h24m42.638016655s) [noise=true , address=81.171.14.70:4031 , piece_id=S4T3ORC533X2OW2Y3BQDAFSYXBTD3THLREQMMJ2G7QPM3UDGHRCQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.860649744 (74h24m42.731732789s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.874184617 (74h24m42.745267652s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.254322126 (74h24m44.125405172s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.671496219 (74h24m44.542579254s) [noise=true , address=81.171.14.70:4025 , piece_id=YX5EYUXYW4ZY3L6CH673H5BSHKUEKTUGGKVSFVENUVJBF3ONZUKA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.750460513 (74h24m44.621543558s) [noise=true , address=81.171.14.70:4033 , piece_id=LSIJ6NCTBALZH2QRPFFTLOXKKWF7J2E3NL2GIYXCDF2RC5KMNBPA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.792368875 (74h24m44.66345192s) [noise=true , address=81.171.14.70:4027 , piece_id=X53DOOCXSAHI64TN5GNSEOHBJK7GL6TLL6W7NJM5KQIVW3ITZ2BA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:45.851581290 (74h24m45.722664326s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.587984219 (74h24m46.459067264s) [noise=true , address=89.149.201.147:4022 , piece_id=ZEX4GP43WCDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.808512099 (74h24m46.679595144s) [noise=true , address=81.171.14.70:4037 , piece_id=MHNTZCXG6COLI7J45SSYK6SZDVSSEBXOFFY7KEGP7NJUDEVLX7NA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:47.152614810 (74h24m47.023697845s) [noise=true , address=81.171.14.70:4038 , piece_id=VFW2FCTINXQNF3AFAFQW5AYWC2EZ2P52ERVUFU5DMH7672AWAQTA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686376464 (74h24m49.557459499s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686397555 (74h24m49.55748059s) [seg_pos={0 0}, upload=65114]: Segment reached optimal threshold of 26 pieces. Dec 8 17:53:49.686927403 (74h24m49.558010438s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: read tcp 188.166.225.170:60652->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687035098 (74h24m49.558118143s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687151320 (74h24m49.558234365s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8): protocol: read tcp 188.166.225.170:58886->89.149.201.147:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687225261 (74h24m49.558308326s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687301826 (74h24m49.558384872s) [seg_pos={0 0}, upload=65114]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 17:53:49.687437044 (74h24m49.558520089s) [upload=65114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:53:49.687467202 (74h24m49.558550247s) [upload=65114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 18:01:19.776524628 (74h32m19.647607673s) [upload=65114]: Done uploading segments... Dec 8 18:01:19.776620922 (74h32m19.647703967s) [upload=65114]: Upload finished. err: metaclient: context deadline exceeded Duration: 18.539090685s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X.........................................X..X..........] duration: 3.377122462s trace: https://jaeger.dev.storj.io/trace/2a639037b48061f3 FAIL (2025-12-08 18:06:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.377166207s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X......................X..] duration: 5.683415689s trace: https://jaeger.dev.storj.io/trace/ab29194e90e610a FAIL (2025-12-08 18:06:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.683452691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.989322533s trace: https://jaeger.dev.storj.io/trace/54ea49bbbba7a774 FAIL (2025-12-08 18:05:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.989354044s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.....................X........X....] duration: 5.074053748s trace: https://jaeger.dev.storj.io/trace/355abf85f96191de FAIL (2025-12-08 18:05:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.074076963s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X......................X.] duration: 5.932193211s trace: https://jaeger.dev.storj.io/trace/ab29194e90e610a FAIL (2025-12-08 18:05:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.93220849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X......................X............X.......................] duration: 5.950410902s trace: https://jaeger.dev.storj.io/trace/9e19cf33ba5771 FAIL (2025-12-08 18:05:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.95044245s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.807396775s trace: https://jaeger.dev.storj.io/trace/531657654e46cf37 FAIL (2025-12-08 18:05:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 17:28:50.970338987 (73h59m50.841422032s) [upload=64765]: Starting upload Dec 8 17:28:50.970430091 (73h59m50.841513126s) [upload=64765]: Done starting upload Dec 8 17:28:50.970493993 (73h59m50.841577028s) [upload=64765]: Uploading segments... Dec 8 17:28:50.970678284 (73h59m50.841761329s) [upload=64765]: Got next segment. Inline: false Dec 8 17:28:50.970713050 (73h59m50.841796085s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:28:50.970718290 (73h59m50.841801325s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:28:53.908198011 (73h59m53.779281066s) [seg_pos={0 0}, upload=64765]: Begin upload segment... Dec 8 17:28:53.908303301 (73h59m53.779386346s) [seg_pos={0 0}, upload=64765]: Done begin upload segment. Dec 8 17:28:53.908383754 (73h59m53.779466809s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.908451002 (73h59m53.779534037s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.909725039 (73h59m53.780808084s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910250399 (73h59m53.781333444s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910344037 (73h59m53.781427082s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910815505 (73h59m53.78189855s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.910826656 (73h59m53.781909701s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911206860 (73h59m53.782289905s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.911517972 (73h59m53.782601007s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912335068 (73h59m53.783418114s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.912943617 (73h59m53.784026652s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913404294 (73h59m53.784487339s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.913787954 (73h59m53.784870989s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914188838 (73h59m53.785271873s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.914646670 (73h59m53.785729715s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:53.915001425 (73h59m53.78608446s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Uploading piece... Dec 8 17:28:55.532728995 (73h59m55.40381204s) [upload=64765]: Next returned nil segment Dec 8 17:28:55.532741217 (73h59m55.403824262s) [upload=64765]: Waiting for error group managing segments... Dec 8 17:28:55.915736688 (73h59m55.786819723s) [noise=true , address=145.79.227.42:4122 , piece_id=MLM6QPPJ2MCQENA5HFEUITX4K5EDLRT2GFIX3EJ2P2ZRTDTJUBMA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.920711762 (73h59m55.791794807s) [noise=true , address=145.79.227.27:4690 , piece_id=CKXE4LIFWOTDHNOYMZ5EN4OG6GKOGJ4PLGNS7C6LVMKFGDIWGIQQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923297686 (73h59m55.794380721s) [noise=true , address=145.79.227.9:4366 , piece_id=7CCZEEQA5JLBM6V3B2D5I2BKMEK7VUK7VHP753FQWJGTGWDG2OWQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.923942233 (73h59m55.795025268s) [noise=true , address=145.79.227.23:4545 , piece_id=WEYYGGO5MT5Q3VIRJZJUIOLY6M2TK7NFKLTFXHELL6SB7USJZHPA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.924659339 (73h59m55.795742384s) [noise=true , address=145.79.227.22:4511 , piece_id=AZPDP7AROHFYMEHGA3A5SGLOSSXMCEBUTOW36PNV32UULIXK5TSA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.926158984 (73h59m55.797242019s) [noise=true , address=145.79.227.32:4835 , piece_id=I7VMKIU2XQILDWOPCGXQXH4CHWVIWQOO2TKPTZLIZ45MHSZR4SZQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.939309806 (73h59m55.810392851s) [noise=true , address=145.79.227.39:4015 , piece_id=IGM7OOCABI4WNE7L4FVGYUB7Z3QDGBLJPYZVUCK3PQFMB27NRWAA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.942475534 (73h59m55.813558579s) [noise=true , address=145.79.227.43:4159 , piece_id=RTS54ZSMOA2IOLGRAAR325ZTLU2PIT3RDWRFPODBE5ATGQTLYFJA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.943183092 (73h59m55.814266137s) [noise=true , address=145.79.227.23:4547 , piece_id=6IBSZVGJTNXSHJHTVLDLLX5PBREDLMSBH5LCRUFM27OWVEVVJOAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.945510774 (73h59m55.816593819s) [noise=true , address=145.79.227.28:4725 , piece_id=EFNPE6656NFVMWYVCVMRMIWUJFQH3YOAOTPDCYRLF4TKCTSG5LAQ, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.948320875 (73h59m55.81940392s) [noise=true , address=145.79.227.29:4941 , piece_id=6AD6ZBP7CQWEOK6W52V3MAJ5RTJSKJGLVYI3ZZ7QEAWOYNZFK2XA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.959845671 (73h59m55.830928716s) [noise=true , address=145.79.227.20:4438 , piece_id=D337IRGCEDDFBJCIDGEIMNXU5GSWBHKALU7JOBNSG52FMVWLWHNA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967292861 (73h59m55.838375897s) [noise=true , address=145.79.227.24:4582 , piece_id=C6L66NXQZGP2VDAJACP7VC6HDVDWGQF4OMLJRXL4JBY7225RQ6SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: <nil> Dec 8 17:28:55.967316656 (73h59m55.838399691s) [seg_pos={0 0}, upload=64765]: Segment reached optimal threshold of 13 pieces. Dec 8 17:28:55.967569218 (73h59m55.838652263s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967576762 (73h59m55.838659797s) [noise=true , address=77.90.31.14:4663 , piece_id=UX73G3Z74CKSZJRUHFCVECKOGYTGX3TLMI6L254IJ676L2IWA3SA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967775601 (73h59m55.838858636s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967794346 (73h59m55.838877391s) [noise=true , address=77.90.31.18:4806 , piece_id=WRXEEQQTK5FFJU554RBOVDED43JP5RM7GLRU4QGV6FJ2THD6A4UA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967805468 (73h59m55.838888513s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 8 17:28:55.967813323 (73h59m55.838896358s) [noise=true , address=77.90.31.10:4517 , piece_id=BVAFMWNACIL7VXDXV3M6V67N2EH3QBFNJX3QCFDAQVZR44Z2KYZA, seg_pos={0 0}, upload=64765]: Long tail cancellation due to optimal threshold Dec 8 17:28:55.967835985 (73h59m55.83891902s) [seg_pos={0 0}, upload=64765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 17:28:55.967920156 (73h59m55.839003201s) [upload=64765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:28:55.967927219 (73h59m55.839010254s) [upload=64765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:29:10.551413142 (74h0m10.422496177s) [upload=64765]: Done uploading segments... Dec 8 17:29:10.551632780 (74h0m10.422715825s) [upload=64765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.807427364s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X......................X] duration: 1m0.00066875s trace: https://jaeger.dev.storj.io/trace/ab29194e90e610a FAIL (2025-12-08 18:05:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00069029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.852080632s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 18:05:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.852106953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.045145602s trace: https://jaeger.dev.storj.io/trace/54ea49bbbba7a774 FAIL (2025-12-08 18:05:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.045175118s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.811532912s trace: https://jaeger.dev.storj.io/trace/2bd986c05c15b190 FAIL (2025-12-08 18:05:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== CDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.842730397 (74h24m40.713813432s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.843290634 (74h24m40.714373669s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.845386525 (74h24m40.716469571s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.846892383 (74h24m40.717975428s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.848772573 (74h24m40.719855618s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.850191745 (74h24m40.72127478s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.851609155 (74h24m40.7226922s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.852832574 (74h24m40.723915619s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:40.854147148 (74h24m40.725230203s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Uploading piece... Dec 8 17:53:41.670064128 (74h24m41.541147173s) [upload=65114]: Next returned nil segment Dec 8 17:53:41.670145664 (74h24m41.541228709s) [upload=65114]: Waiting for error group managing segments... Dec 8 17:53:42.057773064 (74h24m41.928856099s) [noise=true , address=185.167.204.206:4065 , piece_id=7GQAOZEZ6OJKW3UK7U3NXYZM7VXGPHKW6UKHSTEUKJDNRZAYUA4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058200909 (74h24m41.929283954s) [noise=true , address=185.167.204.206:4045 , piece_id=6O2I5IIAPPQFG66OPIOVY2GOHO4C5CM2UXMJHGY45WGQVZK3JGPQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.058523583 (74h24m41.929606628s) [noise=true , address=185.167.204.205:4034 , piece_id=6JYH3N55MJSMXE6VNGSZK7DZX7KUV7CPELNHIMPSFOXG4U2NXRTQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.060741817 (74h24m41.931824863s) [noise=true , address=185.167.204.205:4025 , piece_id=2MDDUA2MUJ6MXICI5H5O4KPAIRJUL7QLBIC4L6MIEKNDYQIO2EJQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.062816829 (74h24m41.933899864s) [noise=true , address=185.167.204.206:4072 , piece_id=X5XLPXKRFRIEQ32PZKY23AS4CFBXSC5NVCJF24IY2LQH4KLGG6RA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.064906518 (74h24m41.935989563s) [noise=true , address=185.167.204.206:4059 , piece_id=KPZWLMWMZA7GCT6JWIMSMLGUW6UVJ3FPWJKMSHP72C42TH5Z4ITA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.065603074 (74h24m41.93668611s) [noise=true , address=185.167.204.206:4047 , piece_id=4G5HDKWP26ER64KVLUXJ2IMQJUXMQ4YLNBYQAMY7MQMGE2PSREEA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.066790916 (74h24m41.937873951s) [noise=true , address=185.167.204.205:4017 , piece_id=2TUWBAIWW6E7GORGD6MBNQFIXUHTD2DTCOVBRJCJW4BFTOPLONCA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.075608666 (74h24m41.946691701s) [noise=true , address=185.167.204.206:4064 , piece_id=2WY55HXXR5LPAR6XSD4GYYYA5BV3A2DPQ5LCPOL3UAWM4IU4ZIJA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.076342644 (74h24m41.947425679s) [noise=true , address=185.167.204.205:4036 , piece_id=FKH55HUCJTNREM6PBSFGWURF5HIGWAM3U6OPQQVTRGEUA6K7JM2Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.077612843 (74h24m41.948695888s) [noise=true , address=185.167.204.205:4032 , piece_id=NF4LIZE2CVGDLBP5O4TFDTJBINY76W2L6ARPSVV5AJXBREKTQS7Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.078994745 (74h24m41.95007778s) [noise=true , address=185.167.204.206:4046 , piece_id=SDP7J6VHYTCJSKOHR4AXONUYML7VNYQIVATGTPENHWS4F4CO3A4A, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.082062506 (74h24m41.953145541s) [noise=true , address=185.167.204.206:4070 , piece_id=5ETK46YMRH46TMYDEYR4CPV3XDF3FS5MR5RSOYPZKD33EXMS5TQQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.096426879 (74h24m41.967509924s) [noise=true , address=185.167.204.206:4049 , piece_id=7IYDBS2GFRUWYC5O62HTXPOL5IPXQ7NZPE4WHDBN5AEAEPECH2VA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.766933610 (74h24m42.638016655s) [noise=true , address=81.171.14.70:4031 , piece_id=S4T3ORC533X2OW2Y3BQDAFSYXBTD3THLREQMMJ2G7QPM3UDGHRCQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.860649744 (74h24m42.731732789s) [noise=true , address=81.171.14.70:4029 , piece_id=3QQWIZOHYA4MRXNV37TOPRD3WWE3V45D2M474ADRLUHB47FLFUIQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:42.874184617 (74h24m42.745267652s) [noise=true , address=81.171.14.70:4035 , piece_id=ULGXVDR7SYRD5FW5B52QXGXQRYUPU3FT4QFSCHWQDXRJ2JM5725Q, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.254322126 (74h24m44.125405172s) [noise=true , address=89.149.201.147:4010 , piece_id=XGAAAQTHKLOR3YLIQW3PKYQGKLES5ZHUHYMDHXFVQWELBQCLYFOQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.671496219 (74h24m44.542579254s) [noise=true , address=81.171.14.70:4025 , piece_id=YX5EYUXYW4ZY3L6CH673H5BSHKUEKTUGGKVSFVENUVJBF3ONZUKA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.750460513 (74h24m44.621543558s) [noise=true , address=81.171.14.70:4033 , piece_id=LSIJ6NCTBALZH2QRPFFTLOXKKWF7J2E3NL2GIYXCDF2RC5KMNBPA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:44.792368875 (74h24m44.66345192s) [noise=true , address=81.171.14.70:4027 , piece_id=X53DOOCXSAHI64TN5GNSEOHBJK7GL6TLL6W7NJM5KQIVW3ITZ2BA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:45.851581290 (74h24m45.722664326s) [noise=true , address=81.171.14.70:4036 , piece_id=XBKXRAUGGAQVV62WSVGNPKXILMXBTIN2VWIAHJM37XLWLUSXWHNA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.587984219 (74h24m46.459067264s) [noise=true , address=89.149.201.147:4022 , piece_id=ZEX4GP43WCDZRW6LEEY46BBQFTFZE5DYUXXXHUTIN3CVN63K65VQ, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:46.808512099 (74h24m46.679595144s) [noise=true , address=81.171.14.70:4037 , piece_id=MHNTZCXG6COLI7J45SSYK6SZDVSSEBXOFFY7KEGP7NJUDEVLX7NA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:47.152614810 (74h24m47.023697845s) [noise=true , address=81.171.14.70:4038 , piece_id=VFW2FCTINXQNF3AFAFQW5AYWC2EZ2P52ERVUFU5DMH7672AWAQTA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686376464 (74h24m49.557459499s) [noise=true , address=89.149.201.147:4021 , piece_id=NX5FUM4724BSTFCEKRLLARMVFV7BFIXKT66XZ6RLIZTPMPWXN2AA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: <nil> Dec 8 17:53:49.686397555 (74h24m49.55748059s) [seg_pos={0 0}, upload=65114]: Segment reached optimal threshold of 26 pieces. Dec 8 17:53:49.686927403 (74h24m49.558010438s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: read tcp 188.166.225.170:60652->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687035098 (74h24m49.558118143s) [noise=true , address=89.149.201.147:4009 , piece_id=7SFNKC45B4BXNTU6IWIETH6P2JXVCCSO2ICFUBDFATHIU3NC4NOA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687151320 (74h24m49.558234365s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8): protocol: read tcp 188.166.225.170:58886->89.149.201.147:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 17:53:49.687225261 (74h24m49.558308326s) [noise=true , address=89.149.201.147:4001 , piece_id=4HDC333BXRRPHPLSYQIKPP5UW2A26C776YDY2GFTQKSFI76SSLGA, seg_pos={0 0}, upload=65114]: Long tail cancellation due to optimal threshold Dec 8 17:53:49.687301826 (74h24m49.558384872s) [seg_pos={0 0}, upload=65114]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 17:53:49.687437044 (74h24m49.558520089s) [upload=65114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:53:49.687467202 (74h24m49.558550247s) [upload=65114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 18:01:19.776524628 (74h32m19.647607673s) [upload=65114]: Done uploading segments... Dec 8 18:01:19.776620922 (74h32m19.647703967s) [upload=65114]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.811561496s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.931272375s trace: https://jaeger.dev.storj.io/trace/43861557210ecf10 FAIL (2025-12-08 18:05:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570401242 (74h10m46.441484277s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.570992959 (74h10m46.442075994s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.571794164 (74h10m46.44287721s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.572722953 (74h10m46.443805988s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573089021 (74h10m46.444172066s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573395113 (74h10m46.444478148s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.573770548 (74h10m46.444853583s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574168597 (74h10m46.445251632s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.574480340 (74h10m46.445563385s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575571548 (74h10m46.446654583s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.575804141 (74h10m46.446887176s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.577142759 (74h10m46.448225795s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.579371745 (74h10m46.45045478s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.581046063 (74h10m46.452129108s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.583365509 (74h10m46.454448554s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585117105 (74h10m46.45620014s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:46.585558214 (74h10m46.456641259s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Uploading piece... Dec 8 17:39:48.038689630 (74h10m47.909772676s) [upload=64919]: Next returned nil segment Dec 8 17:39:48.038704590 (74h10m47.909787635s) [upload=64919]: Waiting for error group managing segments... Dec 8 17:39:48.458801951 (74h10m48.329884996s) [noise=true , address=145.79.227.18:4018 , piece_id=R26COZABHTH7ZLDX5MJCVRFUGUBGC4RQES7JFO6JUC2PZR5SDSNA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.477871734 (74h10m48.348954769s) [noise=true , address=64.49.13.26:4950 , piece_id=52EOZE6XN6I6NKCUROKTGC5EURD4VZ5AGSBUYVZVXG2BPXWYWUKQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.504354151 (74h10m48.375437186s) [noise=true , address=145.79.227.19:4094 , piece_id=S2ILR5FXS5UQWADP5QUR6S3NOU2TZLY6IB3CBDHIKWZIKX4UFBDQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.514794262 (74h10m48.385877307s) [noise=true , address=145.79.227.27:4698 , piece_id=G2MLGDR5APSZHBYJUZEIQ422CNUSISIL42HB6HRKMBQ6Z2LV5C2Q, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.519011292 (74h10m48.390094337s) [noise=true , address=145.79.227.33:4892 , piece_id=Y6EIK6JKD633HHU5TYVZIT3BF2NNOBZMK2RDE2X4VGEG5PKN7W3A, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.572392419 (74h10m48.443475464s) [noise=true , address=77.90.31.17:4803 , piece_id=HRJZI2M6MZDOZZRSTZAHFYB27JUSIYRAATAWMJ2NRN77WWNTIVGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.587308292 (74h10m48.458391337s) [noise=true , address=77.90.31.21:4887 , piece_id=3PGMNHMUOIG3JKZJRYOEJFGY2JLNK6B67WUAUWENOIWIGYJX2ODA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.614491013 (74h10m48.485574058s) [noise=true , address=64.49.13.138:4490 , piece_id=5EUBXNDQHQXJLQ7DNS3Q5LVHSWHBTZCZGAYJGEDEBSD2LES3RLCQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.629307186 (74h10m48.500390231s) [noise=true , address=45.91.116.10:4107 , piece_id=TGXEBFTHJBYEDNGYGVPZKHRVIML3OC3YWZCLZG4RVGXOH5TU2FGQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.634024990 (74h10m48.505108035s) [noise=true , address=77.90.31.16:4762 , piece_id=J3NJB2QZU5WCPWJMORKZVA6LJGZYAFQRHUSMRV4JPJTFG6ACM4WA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.638043983 (74h10m48.509127028s) [noise=true , address=45.91.116.13:4163 , piece_id=QTPSLJ6DI4VPIXHEY3COPLNLPDOBHFGOI55ACHYFFYCUTKI3OVHA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.648228316 (74h10m48.519311361s) [noise=true , address=77.90.31.19:4857 , piece_id=JQCTEHR4X7FVTHMILHXG4FBCQCNA2DRE6KTLXNHQIWYV52BZVRJA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.659302284 (74h10m48.530385329s) [noise=true , address=45.91.116.7:4025 , piece_id=UWM4PKHPAQLTJVFKZUEV4EYKVQ74NL5MWSKOE6QW6DY3VCUXIDEQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663339431 (74h10m48.534422466s) [noise=true , address=77.90.31.14:4684 , piece_id=36HSL6NS2EU456FOHSALR2VLAKOJS6YIZFIA2RLFUDABKWWV4JSQ, seg_pos={0 0}, upload=64919]: Done uploading piece. err: <nil> Dec 8 17:39:48.663362214 (74h10m48.534445259s) [seg_pos={0 0}, upload=64919]: Segment reached optimal threshold of 14 pieces. Dec 8 17:39:48.663607972 (74h10m48.534691017s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663660092 (74h10m48.534743137s) [noise=true , address=77.90.31.18:4831 , piece_id=4DKTWGMPMIICS7JDKOQKMWBG3OIC2X5K6LJYE7HWFXV5MAVXUEMA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663712751 (74h10m48.534795807s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663732810 (74h10m48.534815865s) [noise=true , address=45.91.116.10:4115 , piece_id=3Z7M56K4HSFIFTY6GU6BSYTWDP7DYLFHSOQWMHMAQPFBJXRW7NRA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.663975081 (74h10m48.535058137s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.663985591 (74h10m48.535068636s) [noise=true , address=45.91.116.9:4269 , piece_id=AKHPGJX5MWLBWOHM4YIBSRUYL2BPBIWALVVKM7H3ZWPTZ4URAXDA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664016981 (74h10m48.535100026s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PV6YT6Xdr61PcyNzJsocXBh2NTMtfpqDT9DX49QGi5EiKATJF): protocol: expected piece hash; context canceled; EOF Dec 8 17:39:48.664026439 (74h10m48.535109474s) [noise=true , address=45.91.116.11:4126 , piece_id=UZFWT4IF2VVJIT2Y44I2Q5KORRNTJGLMIMDSSTVKCZOJQVOJGWWA, seg_pos={0 0}, upload=64919]: Long tail cancellation due to optimal threshold Dec 8 17:39:48.664045215 (74h10m48.53512825s) [seg_pos={0 0}, upload=64919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 17:39:48.664116230 (74h10m48.535199265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 17:39:48.664121230 (74h10m48.535204265s) [upload=64919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 17:43:24.723832033 (74h14m24.594915078s) [upload=64919]: Done uploading segments... Dec 8 17:43:24.723896185 (74h14m24.59497923s) [upload=64919]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.931307602s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)