Orbiter History

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [......XX....................................................] duration: 26.274873ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/2af55367efa5ea5e FAIL (2025-12-08 05:02:49 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 26.299741ms edge linksharing bgp ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X............X.X............................................] duration: 3.45305564s trace: https://jaeger.dev.storj.io/trace/498ca12c4376e6b3 FAIL (2025-12-08 05:02:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45314506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X..X..........X.................] duration: 6.286374263s trace: https://jaeger.dev.storj.io/trace/327d8af62af2ad4a FAIL (2025-12-08 05:02:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 04:41:10.986572683 (61h12m10.857655728s) [upload=53698]: Starting upload Dec 8 04:41:10.986697500 (61h12m10.857780545s) [upload=53698]: Done starting upload Dec 8 04:41:10.986782503 (61h12m10.857865538s) [upload=53698]: Uploading segments... Dec 8 04:41:10.987063708 (61h12m10.858146753s) [upload=53698]: Got next segment. Inline: false Dec 8 04:41:10.987116498 (61h12m10.858199543s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 04:41:10.987121758 (61h12m10.858204803s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 04:41:17.462876699 (61h12m17.333959744s) [seg_pos={0 0}, upload=53698]: Begin upload segment... Dec 8 04:41:17.462933336 (61h12m17.334016381s) [seg_pos={0 0}, upload=53698]: Done begin upload segment. Dec 8 04:41:17.462963864 (61h12m17.334046909s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464241417 (61h12m17.335324452s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464758772 (61h12m17.335841817s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465245028 (61h12m17.336328063s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465727617 (61h12m17.336810672s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.466300738 (61h12m17.337383783s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.467904602 (61h12m17.338987647s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.469643584 (61h12m17.340726629s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.470720414 (61h12m17.341803449s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.472841653 (61h12m17.343924708s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.474783842 (61h12m17.345866877s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.478925869 (61h12m17.350008904s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.482146932 (61h12m17.353229977s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.484118316 (61h12m17.355201361s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.486216912 (61h12m17.357299957s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.487427918 (61h12m17.358510953s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:19.196950317 (61h12m19.068033352s) [upload=53698]: Next returned nil segment Dec 8 04:41:19.196972289 (61h12m19.068055324s) [upload=53698]: Waiting for error group managing segments... Dec 8 04:41:19.470957375 (61h12m19.34204042s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.478965694 (61h12m19.350048739s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.488459723 (61h12m19.359542758s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.491834840 (61h12m19.362917885s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.496467461 (61h12m19.367550506s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.497643140 (61h12m19.368726185s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.513620774 (61h12m19.384703809s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.519169180 (61h12m19.390252225s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.582513759 (61h12m19.453596794s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.600844513 (61h12m19.471927549s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.627512986 (61h12m19.498596031s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.923827261 (61h12m19.794910306s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940676124 (61h12m19.811759159s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940696483 (61h12m19.811779518s) [seg_pos={0 0}, upload=53698]: Segment reached optimal threshold of 13 pieces. Dec 8 04:41:19.940920650 (61h12m19.812003695s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.940927292 (61h12m19.812010347s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941137082 (61h12m19.812220117s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941146079 (61h12m19.812229124s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941154806 (61h12m19.812237851s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941162220 (61h12m19.812245255s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941203298 (61h12m19.812286343s) [seg_pos={0 0}, upload=53698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 04:41:19.941272510 (61h12m19.812355555s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:41:19.941277359 (61h12m19.812360404s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:50:23.376780170 (61h21m23.247863205s) [upload=53698]: Done uploading segments... Dec 8 04:50:23.376921268 (61h21m23.248004313s) [upload=53698]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.286402527s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.305290723s trace: https://jaeger.dev.storj.io/trace/4acac71edcec4b82 FAIL (2025-12-08 05:02:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088365227 (61h22m16.959448272s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088855982 (61h22m16.959939017s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088942767 (61h22m16.960025813s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.089748512 (61h22m16.960831557s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.090080424 (61h22m16.961163469s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091013550 (61h22m16.962096595s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091792825 (61h22m16.96287587s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092319577 (61h22m16.963402613s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092355025 (61h22m16.96343806s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.095973455 (61h22m16.9670565s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.910383137 (61h22m17.781466182s) [upload=53840]: Next returned nil segment Dec 8 04:51:17.910461035 (61h22m17.781544071s) [upload=53840]: Waiting for error group managing segments... Dec 8 04:51:18.305376175 (61h22m18.176459221s) [noise=true , address=185.167.204.205:4030 , piece_id=ZHM2LCLRH2HGHUTQZLNHS5VN4SHQUHEXFLKSSG5VDO2O2U562TMA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.305953454 (61h22m18.177036499s) [noise=true , address=185.167.204.205:4025 , piece_id=4UDNENFCOE4JJ2EFKPSHEQQNX5FAWSH6DC3APZWRYSZWLCUCDCXA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307083516 (61h22m18.178166562s) [noise=true , address=185.167.204.205:4010 , piece_id=GTDYB6NWLDINXDPF7YDSXX3JPCEFX2CNVRX5567KDNMYQMZVWFCA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307746278 (61h22m18.178829313s) [noise=true , address=185.167.204.205:4006 , piece_id=QFJ66UCPW2MUX3OUZ64CQLIALWE6LRNOOFFVNMCAPXIXBLNGBQAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.308094291 (61h22m18.179177336s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.309637910 (61h22m18.180720945s) [noise=true , address=185.167.204.206:4043 , piece_id=EBRDIPWFGIK6BKYS5NWMWCQNRDTAO3AHDIFLRVKSR7HHJIS3PZBQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.316825457 (61h22m18.187908502s) [noise=true , address=185.167.204.206:4068 , piece_id=WOUIXUAMIW7OTF2YNM5SAAKWRZLEDAASLVRV3CIAQWRH7F7WKKDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.318376030 (61h22m18.189459075s) [noise=true , address=185.167.204.205:4013 , piece_id=WGR4SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.319751539 (61h22m18.190834574s) [noise=true , address=185.167.204.206:4050 , piece_id=TZO4WU4RMPPNA5ME525DJ34O3YBGLTRYFFWXRRDMPCRTO6LTFVDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.323889458 (61h22m18.194972503s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324447421 (61h22m18.195530466s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324906244 (61h22m18.195989289s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.886193526 (61h22m18.757276571s) [noise=true , address=81.171.14.70:4029 , piece_id=GEDBQ6CWAOJ67P6J6S7LD4EWZRYZV2A6N5V2F4A3HR66L6FERXNQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.896622956 (61h22m18.767706001s) [noise=true , address=81.171.14.70:4033 , piece_id=KVJN4PTIF46QANQMEX4WFBGN3DSOK3GU4B5VSX366ZXHLVNBAEAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.934515209 (61h22m18.805598244s) [noise=true , address=81.171.14.70:4045 , piece_id=J64JUTEWTUGOYDNW4PRLZYZZ24OSZSM6UZJQ7LGJLHTXCXRMSNZA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.940162311 (61h22m18.811245357s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.980483741 (61h22m18.851566776s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.308222254 (61h22m19.179305289s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.493269087 (61h22m19.364352132s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.546757438 (61h22m19.417840473s) [noise=true , address=89.149.201.147:4003 , piece_id=4LLUR2ZGUCDHESM7VRUFMOWDHOEOPJG7CVPOMOP2PUXH5Y6GMVUQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.561727113 (61h22m19.432810158s) [noise=true , address=89.149.201.147:4017 , piece_id=7KT2CZZXUVI6O5IO5KT2DN7VWYRMRMUDAQ4GVU4DZJWTHBE6RCIA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.574035961 (61h22m19.445118996s) [noise=true , address=89.149.201.147:4007 , piece_id=LZD4RYTCP2LPETVP5IHQSYPOS3ESASCFHSFWCFPXFOOUVAIJOAOA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.580254873 (61h22m19.451337918s) [noise=true , address=89.149.201.147:4011 , piece_id=AJRFRVNCSLZAHEVEDDQ5PRYPBVQDKZY6H7SR2HB7ZE4NMTG5AWEQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.619120290 (61h22m19.490203335s) [noise=true , address=89.149.201.147:4015 , piece_id=5QD6RF7RUQRCQTQFGLOKJKO6ULYJETBIC5NGRHB4WO3K7SSSELQQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.874085720 (61h22m19.745168765s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895800389 (61h22m19.766883434s) [noise=true , address=89.149.201.147:4020 , piece_id=QAYH5YBVG52KRKGV5GGDLAY2ZQGXBUL7V2IC2ZJYIALAFCSJRWCQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895827541 (61h22m19.766910596s) [seg_pos={0 0}, upload=53840]: Segment reached optimal threshold of 26 pieces. Dec 8 04:51:19.896083378 (61h22m19.767166413s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896090823 (61h22m19.767173858s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896231440 (61h22m19.767314475s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896243242 (61h22m19.767326278s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896296854 (61h22m19.767379889s) [seg_pos={0 0}, upload=53840]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 04:51:19.896407245 (61h22m19.76749028s) [upload=53840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:51:19.896415951 (61h22m19.767498986s) [upload=53840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:54:58.203928113 (61h25m58.075011158s) [upload=53840]: Done uploading segments... Dec 8 04:54:58.203975553 (61h25m58.075058588s) [upload=53840]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.3053208s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 812.415735ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4679b6ba9d140ac6 FAIL (2025-12-08 05:02:21 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 812.444489ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X..X..........X................] duration: 5.926119172s trace: https://jaeger.dev.storj.io/trace/327d8af62af2ad4a FAIL (2025-12-08 05:02:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 04:41:10.986572683 (61h12m10.857655728s) [upload=53698]: Starting upload Dec 8 04:41:10.986697500 (61h12m10.857780545s) [upload=53698]: Done starting upload Dec 8 04:41:10.986782503 (61h12m10.857865538s) [upload=53698]: Uploading segments... Dec 8 04:41:10.987063708 (61h12m10.858146753s) [upload=53698]: Got next segment. Inline: false Dec 8 04:41:10.987116498 (61h12m10.858199543s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 04:41:10.987121758 (61h12m10.858204803s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 04:41:17.462876699 (61h12m17.333959744s) [seg_pos={0 0}, upload=53698]: Begin upload segment... Dec 8 04:41:17.462933336 (61h12m17.334016381s) [seg_pos={0 0}, upload=53698]: Done begin upload segment. Dec 8 04:41:17.462963864 (61h12m17.334046909s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464241417 (61h12m17.335324452s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464758772 (61h12m17.335841817s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465245028 (61h12m17.336328063s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465727617 (61h12m17.336810672s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.466300738 (61h12m17.337383783s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.467904602 (61h12m17.338987647s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.469643584 (61h12m17.340726629s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.470720414 (61h12m17.341803449s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.472841653 (61h12m17.343924708s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.474783842 (61h12m17.345866877s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.478925869 (61h12m17.350008904s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.482146932 (61h12m17.353229977s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.484118316 (61h12m17.355201361s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.486216912 (61h12m17.357299957s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.487427918 (61h12m17.358510953s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:19.196950317 (61h12m19.068033352s) [upload=53698]: Next returned nil segment Dec 8 04:41:19.196972289 (61h12m19.068055324s) [upload=53698]: Waiting for error group managing segments... Dec 8 04:41:19.470957375 (61h12m19.34204042s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.478965694 (61h12m19.350048739s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.488459723 (61h12m19.359542758s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.491834840 (61h12m19.362917885s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.496467461 (61h12m19.367550506s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.497643140 (61h12m19.368726185s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.513620774 (61h12m19.384703809s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.519169180 (61h12m19.390252225s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.582513759 (61h12m19.453596794s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.600844513 (61h12m19.471927549s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.627512986 (61h12m19.498596031s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.923827261 (61h12m19.794910306s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940676124 (61h12m19.811759159s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940696483 (61h12m19.811779518s) [seg_pos={0 0}, upload=53698]: Segment reached optimal threshold of 13 pieces. Dec 8 04:41:19.940920650 (61h12m19.812003695s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.940927292 (61h12m19.812010347s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941137082 (61h12m19.812220117s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941146079 (61h12m19.812229124s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941154806 (61h12m19.812237851s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941162220 (61h12m19.812245255s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941203298 (61h12m19.812286343s) [seg_pos={0 0}, upload=53698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 04:41:19.941272510 (61h12m19.812355555s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:41:19.941277359 (61h12m19.812360404s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:50:23.376780170 (61h21m23.247863205s) [upload=53698]: Done uploading segments... Dec 8 04:50:23.376921268 (61h21m23.248004313s) [upload=53698]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.926152096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.031226911s trace: https://jaeger.dev.storj.io/trace/56ba3625fd29e75 FAIL (2025-12-08 05:02:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 05:00:42.944581611 (61h31m42.815664646s) [upload=53967]: Starting upload Dec 8 05:00:42.944670691 (61h31m42.815753736s) [upload=53967]: Done starting upload Dec 8 05:00:42.944716888 (61h31m42.815799933s) [upload=53967]: Uploading segments... Dec 8 05:00:42.944796501 (61h31m42.815879556s) [upload=53967]: Got next segment. Inline: true Dec 8 05:00:42.944804315 (61h31m42.81588736s) [upload=53967]: Waiting for error group managing segments... Dec 8 05:00:42.944874950 (61h31m42.815957995s) [upload=53967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:00:42.944882524 (61h31m42.815965569s) [upload=53967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 05:00:42.944886632 (61h31m42.815969677s) [upload=53967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 05:01:41.678299330 (61h32m41.549382375s) [upload=53967]: Done uploading segments... Dec 8 05:01:41.678338003 (61h32m41.549421038s) [upload=53967]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.031392146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.625890526s trace: https://jaeger.dev.storj.io/trace/6506fd1cc11bbd33 FAIL (2025-12-08 05:02:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.626029631s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 705.99818ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4679b6ba9d140ac6 FAIL (2025-12-08 05:01:59 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 706.029219ms core satellite us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [.......XX...................................................] duration: 25.913025ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/2af55367efa5ea5e FAIL (2025-12-08 05:01:51 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 25.940296ms edge linksharing bgp ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X......................X...] duration: 5.917992605s trace: https://jaeger.dev.storj.io/trace/67bd73b777c66f73 FAIL (2025-12-08 05:01:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.91802088s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 481.655196ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/17143a9730f83ede FAIL (2025-12-08 05:01:49 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 481.684422ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X..X..........X...............] duration: 5.99694475s trace: https://jaeger.dev.storj.io/trace/327d8af62af2ad4a FAIL (2025-12-08 05:01:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 04:41:10.986572683 (61h12m10.857655728s) [upload=53698]: Starting upload Dec 8 04:41:10.986697500 (61h12m10.857780545s) [upload=53698]: Done starting upload Dec 8 04:41:10.986782503 (61h12m10.857865538s) [upload=53698]: Uploading segments... Dec 8 04:41:10.987063708 (61h12m10.858146753s) [upload=53698]: Got next segment. Inline: false Dec 8 04:41:10.987116498 (61h12m10.858199543s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 04:41:10.987121758 (61h12m10.858204803s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 04:41:17.462876699 (61h12m17.333959744s) [seg_pos={0 0}, upload=53698]: Begin upload segment... Dec 8 04:41:17.462933336 (61h12m17.334016381s) [seg_pos={0 0}, upload=53698]: Done begin upload segment. Dec 8 04:41:17.462963864 (61h12m17.334046909s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464241417 (61h12m17.335324452s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464758772 (61h12m17.335841817s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465245028 (61h12m17.336328063s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465727617 (61h12m17.336810672s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.466300738 (61h12m17.337383783s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.467904602 (61h12m17.338987647s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.469643584 (61h12m17.340726629s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.470720414 (61h12m17.341803449s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.472841653 (61h12m17.343924708s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.474783842 (61h12m17.345866877s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.478925869 (61h12m17.350008904s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.482146932 (61h12m17.353229977s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.484118316 (61h12m17.355201361s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.486216912 (61h12m17.357299957s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.487427918 (61h12m17.358510953s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:19.196950317 (61h12m19.068033352s) [upload=53698]: Next returned nil segment Dec 8 04:41:19.196972289 (61h12m19.068055324s) [upload=53698]: Waiting for error group managing segments... Dec 8 04:41:19.470957375 (61h12m19.34204042s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.478965694 (61h12m19.350048739s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.488459723 (61h12m19.359542758s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.491834840 (61h12m19.362917885s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.496467461 (61h12m19.367550506s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.497643140 (61h12m19.368726185s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.513620774 (61h12m19.384703809s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.519169180 (61h12m19.390252225s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.582513759 (61h12m19.453596794s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.600844513 (61h12m19.471927549s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.627512986 (61h12m19.498596031s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.923827261 (61h12m19.794910306s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940676124 (61h12m19.811759159s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940696483 (61h12m19.811779518s) [seg_pos={0 0}, upload=53698]: Segment reached optimal threshold of 13 pieces. Dec 8 04:41:19.940920650 (61h12m19.812003695s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.940927292 (61h12m19.812010347s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941137082 (61h12m19.812220117s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941146079 (61h12m19.812229124s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941154806 (61h12m19.812237851s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941162220 (61h12m19.812245255s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941203298 (61h12m19.812286343s) [seg_pos={0 0}, upload=53698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 04:41:19.941272510 (61h12m19.812355555s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:41:19.941277359 (61h12m19.812360404s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:50:23.376780170 (61h21m23.247863205s) [upload=53698]: Done uploading segments... Dec 8 04:50:23.376921268 (61h21m23.248004313s) [upload=53698]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.996974659s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.059393376s trace: https://jaeger.dev.storj.io/trace/56ba3625fd29e75 FAIL (2025-12-08 05:01:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 05:00:42.944581611 (61h31m42.815664646s) [upload=53967]: Starting upload Dec 8 05:00:42.944670691 (61h31m42.815753736s) [upload=53967]: Done starting upload Dec 8 05:00:42.944716888 (61h31m42.815799933s) [upload=53967]: Uploading segments... Dec 8 05:00:42.944796501 (61h31m42.815879556s) [upload=53967]: Got next segment. Inline: true Dec 8 05:00:42.944804315 (61h31m42.81588736s) [upload=53967]: Waiting for error group managing segments... Dec 8 05:00:42.944874950 (61h31m42.815957995s) [upload=53967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:00:42.944882524 (61h31m42.815965569s) [upload=53967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 05:00:42.944886632 (61h31m42.815969677s) [upload=53967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 05:01:41.678299330 (61h32m41.549382375s) [upload=53967]: Done uploading segments... Dec 8 05:01:41.678338003 (61h32m41.549421038s) [upload=53967]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.059407133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001760002s trace: https://jaeger.dev.storj.io/trace/56ba3625fd29e75 FAIL (2025-12-08 05:01:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 05:00:42.944581611 (61h31m42.815664646s) [upload=53967]: Starting upload Dec 8 05:00:42.944670691 (61h31m42.815753736s) [upload=53967]: Done starting upload Dec 8 05:00:42.944716888 (61h31m42.815799933s) [upload=53967]: Uploading segments... Dec 8 05:00:42.944796501 (61h31m42.815879556s) [upload=53967]: Got next segment. Inline: true Dec 8 05:00:42.944804315 (61h31m42.81588736s) [upload=53967]: Waiting for error group managing segments... Dec 8 05:00:42.944874950 (61h31m42.815957995s) [upload=53967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 05:00:42.944882524 (61h31m42.815965569s) [upload=53967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 05:00:42.944886632 (61h31m42.815969677s) [upload=53967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 05:01:41.678299330 (61h32m41.549382375s) [upload=53967]: Done uploading segments... Dec 8 05:01:41.678338003 (61h32m41.549421038s) [upload=53967]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.001782625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X............X.X...........................................] duration: 3.59242197s trace: https://jaeger.dev.storj.io/trace/498ca12c4376e6b3 FAIL (2025-12-08 05:01:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.592447738s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 475.364986ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/17143a9730f83ede FAIL (2025-12-08 05:01:31 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.386748ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 11.750828092s trace: https://jaeger.dev.storj.io/trace/4acac71edcec4b82 FAIL (2025-12-08 05:01:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088365227 (61h22m16.959448272s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088855982 (61h22m16.959939017s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088942767 (61h22m16.960025813s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.089748512 (61h22m16.960831557s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.090080424 (61h22m16.961163469s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091013550 (61h22m16.962096595s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091792825 (61h22m16.96287587s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092319577 (61h22m16.963402613s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092355025 (61h22m16.96343806s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.095973455 (61h22m16.9670565s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.910383137 (61h22m17.781466182s) [upload=53840]: Next returned nil segment Dec 8 04:51:17.910461035 (61h22m17.781544071s) [upload=53840]: Waiting for error group managing segments... Dec 8 04:51:18.305376175 (61h22m18.176459221s) [noise=true , address=185.167.204.205:4030 , piece_id=ZHM2LCLRH2HGHUTQZLNHS5VN4SHQUHEXFLKSSG5VDO2O2U562TMA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.305953454 (61h22m18.177036499s) [noise=true , address=185.167.204.205:4025 , piece_id=4UDNENFCOE4JJ2EFKPSHEQQNX5FAWSH6DC3APZWRYSZWLCUCDCXA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307083516 (61h22m18.178166562s) [noise=true , address=185.167.204.205:4010 , piece_id=GTDYB6NWLDINXDPF7YDSXX3JPCEFX2CNVRX5567KDNMYQMZVWFCA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307746278 (61h22m18.178829313s) [noise=true , address=185.167.204.205:4006 , piece_id=QFJ66UCPW2MUX3OUZ64CQLIALWE6LRNOOFFVNMCAPXIXBLNGBQAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.308094291 (61h22m18.179177336s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.309637910 (61h22m18.180720945s) [noise=true , address=185.167.204.206:4043 , piece_id=EBRDIPWFGIK6BKYS5NWMWCQNRDTAO3AHDIFLRVKSR7HHJIS3PZBQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.316825457 (61h22m18.187908502s) [noise=true , address=185.167.204.206:4068 , piece_id=WOUIXUAMIW7OTF2YNM5SAAKWRZLEDAASLVRV3CIAQWRH7F7WKKDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.318376030 (61h22m18.189459075s) [noise=true , address=185.167.204.205:4013 , piece_id=WGR4SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.319751539 (61h22m18.190834574s) [noise=true , address=185.167.204.206:4050 , piece_id=TZO4WU4RMPPNA5ME525DJ34O3YBGLTRYFFWXRRDMPCRTO6LTFVDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.323889458 (61h22m18.194972503s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324447421 (61h22m18.195530466s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324906244 (61h22m18.195989289s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.886193526 (61h22m18.757276571s) [noise=true , address=81.171.14.70:4029 , piece_id=GEDBQ6CWAOJ67P6J6S7LD4EWZRYZV2A6N5V2F4A3HR66L6FERXNQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.896622956 (61h22m18.767706001s) [noise=true , address=81.171.14.70:4033 , piece_id=KVJN4PTIF46QANQMEX4WFBGN3DSOK3GU4B5VSX366ZXHLVNBAEAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.934515209 (61h22m18.805598244s) [noise=true , address=81.171.14.70:4045 , piece_id=J64JUTEWTUGOYDNW4PRLZYZZ24OSZSM6UZJQ7LGJLHTXCXRMSNZA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.940162311 (61h22m18.811245357s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.980483741 (61h22m18.851566776s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.308222254 (61h22m19.179305289s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.493269087 (61h22m19.364352132s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.546757438 (61h22m19.417840473s) [noise=true , address=89.149.201.147:4003 , piece_id=4LLUR2ZGUCDHESM7VRUFMOWDHOEOPJG7CVPOMOP2PUXH5Y6GMVUQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.561727113 (61h22m19.432810158s) [noise=true , address=89.149.201.147:4017 , piece_id=7KT2CZZXUVI6O5IO5KT2DN7VWYRMRMUDAQ4GVU4DZJWTHBE6RCIA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.574035961 (61h22m19.445118996s) [noise=true , address=89.149.201.147:4007 , piece_id=LZD4RYTCP2LPETVP5IHQSYPOS3ESASCFHSFWCFPXFOOUVAIJOAOA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.580254873 (61h22m19.451337918s) [noise=true , address=89.149.201.147:4011 , piece_id=AJRFRVNCSLZAHEVEDDQ5PRYPBVQDKZY6H7SR2HB7ZE4NMTG5AWEQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.619120290 (61h22m19.490203335s) [noise=true , address=89.149.201.147:4015 , piece_id=5QD6RF7RUQRCQTQFGLOKJKO6ULYJETBIC5NGRHB4WO3K7SSSELQQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.874085720 (61h22m19.745168765s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895800389 (61h22m19.766883434s) [noise=true , address=89.149.201.147:4020 , piece_id=QAYH5YBVG52KRKGV5GGDLAY2ZQGXBUL7V2IC2ZJYIALAFCSJRWCQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895827541 (61h22m19.766910596s) [seg_pos={0 0}, upload=53840]: Segment reached optimal threshold of 26 pieces. Dec 8 04:51:19.896083378 (61h22m19.767166413s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896090823 (61h22m19.767173858s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896231440 (61h22m19.767314475s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896243242 (61h22m19.767326278s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896296854 (61h22m19.767379889s) [seg_pos={0 0}, upload=53840]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 04:51:19.896407245 (61h22m19.76749028s) [upload=53840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:51:19.896415951 (61h22m19.767498986s) [upload=53840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:54:58.203928113 (61h25m58.075011158s) [upload=53840]: Done uploading segments... Dec 8 04:54:58.203975553 (61h25m58.075058588s) [upload=53840]: Upload finished. err: metaclient: context deadline exceeded Duration: 11.750858009s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 704.04058ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4679b6ba9d140ac6 FAIL (2025-12-08 05:01:28 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 704.063574ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X..X..........X..............] duration: 6.459527728s trace: https://jaeger.dev.storj.io/trace/327d8af62af2ad4a FAIL (2025-12-08 05:01:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 04:41:10.986572683 (61h12m10.857655728s) [upload=53698]: Starting upload Dec 8 04:41:10.986697500 (61h12m10.857780545s) [upload=53698]: Done starting upload Dec 8 04:41:10.986782503 (61h12m10.857865538s) [upload=53698]: Uploading segments... Dec 8 04:41:10.987063708 (61h12m10.858146753s) [upload=53698]: Got next segment. Inline: false Dec 8 04:41:10.987116498 (61h12m10.858199543s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 04:41:10.987121758 (61h12m10.858204803s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 04:41:17.462876699 (61h12m17.333959744s) [seg_pos={0 0}, upload=53698]: Begin upload segment... Dec 8 04:41:17.462933336 (61h12m17.334016381s) [seg_pos={0 0}, upload=53698]: Done begin upload segment. Dec 8 04:41:17.462963864 (61h12m17.334046909s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464241417 (61h12m17.335324452s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464758772 (61h12m17.335841817s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465245028 (61h12m17.336328063s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465727617 (61h12m17.336810672s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.466300738 (61h12m17.337383783s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.467904602 (61h12m17.338987647s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.469643584 (61h12m17.340726629s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.470720414 (61h12m17.341803449s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.472841653 (61h12m17.343924708s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.474783842 (61h12m17.345866877s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.478925869 (61h12m17.350008904s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.482146932 (61h12m17.353229977s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.484118316 (61h12m17.355201361s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.486216912 (61h12m17.357299957s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.487427918 (61h12m17.358510953s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:19.196950317 (61h12m19.068033352s) [upload=53698]: Next returned nil segment Dec 8 04:41:19.196972289 (61h12m19.068055324s) [upload=53698]: Waiting for error group managing segments... Dec 8 04:41:19.470957375 (61h12m19.34204042s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.478965694 (61h12m19.350048739s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.488459723 (61h12m19.359542758s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.491834840 (61h12m19.362917885s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.496467461 (61h12m19.367550506s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.497643140 (61h12m19.368726185s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.513620774 (61h12m19.384703809s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.519169180 (61h12m19.390252225s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.582513759 (61h12m19.453596794s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.600844513 (61h12m19.471927549s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.627512986 (61h12m19.498596031s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.923827261 (61h12m19.794910306s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940676124 (61h12m19.811759159s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940696483 (61h12m19.811779518s) [seg_pos={0 0}, upload=53698]: Segment reached optimal threshold of 13 pieces. Dec 8 04:41:19.940920650 (61h12m19.812003695s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.940927292 (61h12m19.812010347s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941137082 (61h12m19.812220117s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941146079 (61h12m19.812229124s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941154806 (61h12m19.812237851s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941162220 (61h12m19.812245255s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941203298 (61h12m19.812286343s) [seg_pos={0 0}, upload=53698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 04:41:19.941272510 (61h12m19.812355555s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:41:19.941277359 (61h12m19.812360404s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:50:23.376780170 (61h21m23.247863205s) [upload=53698]: Done uploading segments... Dec 8 04:50:23.376921268 (61h21m23.248004313s) [upload=53698]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.459553246s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [........XX..................................................] duration: 24.526034ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/2af55367efa5ea5e FAIL (2025-12-08 05:01:05 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 24.559248ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 483.596523ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/17143a9730f83ede FAIL (2025-12-08 05:01:04 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 483.632221ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X..X..........X.............] duration: 5.908401198s trace: https://jaeger.dev.storj.io/trace/327d8af62af2ad4a FAIL (2025-12-08 05:01:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 04:41:10.986572683 (61h12m10.857655728s) [upload=53698]: Starting upload Dec 8 04:41:10.986697500 (61h12m10.857780545s) [upload=53698]: Done starting upload Dec 8 04:41:10.986782503 (61h12m10.857865538s) [upload=53698]: Uploading segments... Dec 8 04:41:10.987063708 (61h12m10.858146753s) [upload=53698]: Got next segment. Inline: false Dec 8 04:41:10.987116498 (61h12m10.858199543s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 04:41:10.987121758 (61h12m10.858204803s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 04:41:17.462876699 (61h12m17.333959744s) [seg_pos={0 0}, upload=53698]: Begin upload segment... Dec 8 04:41:17.462933336 (61h12m17.334016381s) [seg_pos={0 0}, upload=53698]: Done begin upload segment. Dec 8 04:41:17.462963864 (61h12m17.334046909s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464241417 (61h12m17.335324452s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464758772 (61h12m17.335841817s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465245028 (61h12m17.336328063s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465727617 (61h12m17.336810672s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.466300738 (61h12m17.337383783s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.467904602 (61h12m17.338987647s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.469643584 (61h12m17.340726629s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.470720414 (61h12m17.341803449s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.472841653 (61h12m17.343924708s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.474783842 (61h12m17.345866877s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.478925869 (61h12m17.350008904s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.482146932 (61h12m17.353229977s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.484118316 (61h12m17.355201361s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.486216912 (61h12m17.357299957s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.487427918 (61h12m17.358510953s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:19.196950317 (61h12m19.068033352s) [upload=53698]: Next returned nil segment Dec 8 04:41:19.196972289 (61h12m19.068055324s) [upload=53698]: Waiting for error group managing segments... Dec 8 04:41:19.470957375 (61h12m19.34204042s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.478965694 (61h12m19.350048739s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.488459723 (61h12m19.359542758s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.491834840 (61h12m19.362917885s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.496467461 (61h12m19.367550506s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.497643140 (61h12m19.368726185s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.513620774 (61h12m19.384703809s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.519169180 (61h12m19.390252225s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.582513759 (61h12m19.453596794s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.600844513 (61h12m19.471927549s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.627512986 (61h12m19.498596031s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.923827261 (61h12m19.794910306s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940676124 (61h12m19.811759159s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940696483 (61h12m19.811779518s) [seg_pos={0 0}, upload=53698]: Segment reached optimal threshold of 13 pieces. Dec 8 04:41:19.940920650 (61h12m19.812003695s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.940927292 (61h12m19.812010347s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941137082 (61h12m19.812220117s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941146079 (61h12m19.812229124s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941154806 (61h12m19.812237851s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941162220 (61h12m19.812245255s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941203298 (61h12m19.812286343s) [seg_pos={0 0}, upload=53698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 04:41:19.941272510 (61h12m19.812355555s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:41:19.941277359 (61h12m19.812360404s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:50:23.376780170 (61h21m23.247863205s) [upload=53698]: Done uploading segments... Dec 8 04:50:23.376921268 (61h21m23.248004313s) [upload=53698]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.908430964s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X......................X..] duration: 5.776657988s trace: https://jaeger.dev.storj.io/trace/67bd73b777c66f73 FAIL (2025-12-08 05:00:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.776695631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.66417082s trace: https://jaeger.dev.storj.io/trace/6506fd1cc11bbd33 FAIL (2025-12-08 05:00:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.664198573s edge gateway us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [.........XX.................................................] duration: 28.274652ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/2af55367efa5ea5e FAIL (2025-12-08 05:00:48 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 28.297405ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 468.426456ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/17143a9730f83ede FAIL (2025-12-08 05:00:44 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.448878ms core satellite eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 694.731372ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4679b6ba9d140ac6 FAIL (2025-12-08 05:00:37 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 694.884033ms core satellite us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.840560008s trace: https://jaeger.dev.storj.io/trace/4acac71edcec4b82 FAIL (2025-12-08 05:00:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088365227 (61h22m16.959448272s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088855982 (61h22m16.959939017s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088942767 (61h22m16.960025813s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.089748512 (61h22m16.960831557s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.090080424 (61h22m16.961163469s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091013550 (61h22m16.962096595s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091792825 (61h22m16.96287587s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092319577 (61h22m16.963402613s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092355025 (61h22m16.96343806s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.095973455 (61h22m16.9670565s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.910383137 (61h22m17.781466182s) [upload=53840]: Next returned nil segment Dec 8 04:51:17.910461035 (61h22m17.781544071s) [upload=53840]: Waiting for error group managing segments... Dec 8 04:51:18.305376175 (61h22m18.176459221s) [noise=true , address=185.167.204.205:4030 , piece_id=ZHM2LCLRH2HGHUTQZLNHS5VN4SHQUHEXFLKSSG5VDO2O2U562TMA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.305953454 (61h22m18.177036499s) [noise=true , address=185.167.204.205:4025 , piece_id=4UDNENFCOE4JJ2EFKPSHEQQNX5FAWSH6DC3APZWRYSZWLCUCDCXA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307083516 (61h22m18.178166562s) [noise=true , address=185.167.204.205:4010 , piece_id=GTDYB6NWLDINXDPF7YDSXX3JPCEFX2CNVRX5567KDNMYQMZVWFCA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307746278 (61h22m18.178829313s) [noise=true , address=185.167.204.205:4006 , piece_id=QFJ66UCPW2MUX3OUZ64CQLIALWE6LRNOOFFVNMCAPXIXBLNGBQAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.308094291 (61h22m18.179177336s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.309637910 (61h22m18.180720945s) [noise=true , address=185.167.204.206:4043 , piece_id=EBRDIPWFGIK6BKYS5NWMWCQNRDTAO3AHDIFLRVKSR7HHJIS3PZBQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.316825457 (61h22m18.187908502s) [noise=true , address=185.167.204.206:4068 , piece_id=WOUIXUAMIW7OTF2YNM5SAAKWRZLEDAASLVRV3CIAQWRH7F7WKKDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.318376030 (61h22m18.189459075s) [noise=true , address=185.167.204.205:4013 , piece_id=WGR4SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.319751539 (61h22m18.190834574s) [noise=true , address=185.167.204.206:4050 , piece_id=TZO4WU4RMPPNA5ME525DJ34O3YBGLTRYFFWXRRDMPCRTO6LTFVDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.323889458 (61h22m18.194972503s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324447421 (61h22m18.195530466s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324906244 (61h22m18.195989289s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.886193526 (61h22m18.757276571s) [noise=true , address=81.171.14.70:4029 , piece_id=GEDBQ6CWAOJ67P6J6S7LD4EWZRYZV2A6N5V2F4A3HR66L6FERXNQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.896622956 (61h22m18.767706001s) [noise=true , address=81.171.14.70:4033 , piece_id=KVJN4PTIF46QANQMEX4WFBGN3DSOK3GU4B5VSX366ZXHLVNBAEAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.934515209 (61h22m18.805598244s) [noise=true , address=81.171.14.70:4045 , piece_id=J64JUTEWTUGOYDNW4PRLZYZZ24OSZSM6UZJQ7LGJLHTXCXRMSNZA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.940162311 (61h22m18.811245357s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.980483741 (61h22m18.851566776s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.308222254 (61h22m19.179305289s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.493269087 (61h22m19.364352132s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.546757438 (61h22m19.417840473s) [noise=true , address=89.149.201.147:4003 , piece_id=4LLUR2ZGUCDHESM7VRUFMOWDHOEOPJG7CVPOMOP2PUXH5Y6GMVUQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.561727113 (61h22m19.432810158s) [noise=true , address=89.149.201.147:4017 , piece_id=7KT2CZZXUVI6O5IO5KT2DN7VWYRMRMUDAQ4GVU4DZJWTHBE6RCIA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.574035961 (61h22m19.445118996s) [noise=true , address=89.149.201.147:4007 , piece_id=LZD4RYTCP2LPETVP5IHQSYPOS3ESASCFHSFWCFPXFOOUVAIJOAOA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.580254873 (61h22m19.451337918s) [noise=true , address=89.149.201.147:4011 , piece_id=AJRFRVNCSLZAHEVEDDQ5PRYPBVQDKZY6H7SR2HB7ZE4NMTG5AWEQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.619120290 (61h22m19.490203335s) [noise=true , address=89.149.201.147:4015 , piece_id=5QD6RF7RUQRCQTQFGLOKJKO6ULYJETBIC5NGRHB4WO3K7SSSELQQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.874085720 (61h22m19.745168765s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895800389 (61h22m19.766883434s) [noise=true , address=89.149.201.147:4020 , piece_id=QAYH5YBVG52KRKGV5GGDLAY2ZQGXBUL7V2IC2ZJYIALAFCSJRWCQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895827541 (61h22m19.766910596s) [seg_pos={0 0}, upload=53840]: Segment reached optimal threshold of 26 pieces. Dec 8 04:51:19.896083378 (61h22m19.767166413s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896090823 (61h22m19.767173858s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896231440 (61h22m19.767314475s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896243242 (61h22m19.767326278s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896296854 (61h22m19.767379889s) [seg_pos={0 0}, upload=53840]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 04:51:19.896407245 (61h22m19.76749028s) [upload=53840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:51:19.896415951 (61h22m19.767498986s) [upload=53840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:54:58.203928113 (61h25m58.075011158s) [upload=53840]: Done uploading segments... Dec 8 04:54:58.203975553 (61h25m58.075058588s) [upload=53840]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.840591527s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X............X.X..........................................] duration: 3.463563516s trace: https://jaeger.dev.storj.io/trace/498ca12c4376e6b3 FAIL (2025-12-08 05:00:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463722299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.305811817s trace: https://jaeger.dev.storj.io/trace/4acac71edcec4b82 FAIL (2025-12-08 05:00:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088365227 (61h22m16.959448272s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088855982 (61h22m16.959939017s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088942767 (61h22m16.960025813s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.089748512 (61h22m16.960831557s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.090080424 (61h22m16.961163469s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091013550 (61h22m16.962096595s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091792825 (61h22m16.96287587s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092319577 (61h22m16.963402613s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092355025 (61h22m16.96343806s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.095973455 (61h22m16.9670565s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.910383137 (61h22m17.781466182s) [upload=53840]: Next returned nil segment Dec 8 04:51:17.910461035 (61h22m17.781544071s) [upload=53840]: Waiting for error group managing segments... Dec 8 04:51:18.305376175 (61h22m18.176459221s) [noise=true , address=185.167.204.205:4030 , piece_id=ZHM2LCLRH2HGHUTQZLNHS5VN4SHQUHEXFLKSSG5VDO2O2U562TMA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.305953454 (61h22m18.177036499s) [noise=true , address=185.167.204.205:4025 , piece_id=4UDNENFCOE4JJ2EFKPSHEQQNX5FAWSH6DC3APZWRYSZWLCUCDCXA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307083516 (61h22m18.178166562s) [noise=true , address=185.167.204.205:4010 , piece_id=GTDYB6NWLDINXDPF7YDSXX3JPCEFX2CNVRX5567KDNMYQMZVWFCA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307746278 (61h22m18.178829313s) [noise=true , address=185.167.204.205:4006 , piece_id=QFJ66UCPW2MUX3OUZ64CQLIALWE6LRNOOFFVNMCAPXIXBLNGBQAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.308094291 (61h22m18.179177336s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.309637910 (61h22m18.180720945s) [noise=true , address=185.167.204.206:4043 , piece_id=EBRDIPWFGIK6BKYS5NWMWCQNRDTAO3AHDIFLRVKSR7HHJIS3PZBQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.316825457 (61h22m18.187908502s) [noise=true , address=185.167.204.206:4068 , piece_id=WOUIXUAMIW7OTF2YNM5SAAKWRZLEDAASLVRV3CIAQWRH7F7WKKDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.318376030 (61h22m18.189459075s) [noise=true , address=185.167.204.205:4013 , piece_id=WGR4SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.319751539 (61h22m18.190834574s) [noise=true , address=185.167.204.206:4050 , piece_id=TZO4WU4RMPPNA5ME525DJ34O3YBGLTRYFFWXRRDMPCRTO6LTFVDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.323889458 (61h22m18.194972503s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324447421 (61h22m18.195530466s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324906244 (61h22m18.195989289s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.886193526 (61h22m18.757276571s) [noise=true , address=81.171.14.70:4029 , piece_id=GEDBQ6CWAOJ67P6J6S7LD4EWZRYZV2A6N5V2F4A3HR66L6FERXNQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.896622956 (61h22m18.767706001s) [noise=true , address=81.171.14.70:4033 , piece_id=KVJN4PTIF46QANQMEX4WFBGN3DSOK3GU4B5VSX366ZXHLVNBAEAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.934515209 (61h22m18.805598244s) [noise=true , address=81.171.14.70:4045 , piece_id=J64JUTEWTUGOYDNW4PRLZYZZ24OSZSM6UZJQ7LGJLHTXCXRMSNZA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.940162311 (61h22m18.811245357s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.980483741 (61h22m18.851566776s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.308222254 (61h22m19.179305289s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.493269087 (61h22m19.364352132s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.546757438 (61h22m19.417840473s) [noise=true , address=89.149.201.147:4003 , piece_id=4LLUR2ZGUCDHESM7VRUFMOWDHOEOPJG7CVPOMOP2PUXH5Y6GMVUQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.561727113 (61h22m19.432810158s) [noise=true , address=89.149.201.147:4017 , piece_id=7KT2CZZXUVI6O5IO5KT2DN7VWYRMRMUDAQ4GVU4DZJWTHBE6RCIA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.574035961 (61h22m19.445118996s) [noise=true , address=89.149.201.147:4007 , piece_id=LZD4RYTCP2LPETVP5IHQSYPOS3ESASCFHSFWCFPXFOOUVAIJOAOA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.580254873 (61h22m19.451337918s) [noise=true , address=89.149.201.147:4011 , piece_id=AJRFRVNCSLZAHEVEDDQ5PRYPBVQDKZY6H7SR2HB7ZE4NMTG5AWEQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.619120290 (61h22m19.490203335s) [noise=true , address=89.149.201.147:4015 , piece_id=5QD6RF7RUQRCQTQFGLOKJKO6ULYJETBIC5NGRHB4WO3K7SSSELQQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.874085720 (61h22m19.745168765s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895800389 (61h22m19.766883434s) [noise=true , address=89.149.201.147:4020 , piece_id=QAYH5YBVG52KRKGV5GGDLAY2ZQGXBUL7V2IC2ZJYIALAFCSJRWCQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895827541 (61h22m19.766910596s) [seg_pos={0 0}, upload=53840]: Segment reached optimal threshold of 26 pieces. Dec 8 04:51:19.896083378 (61h22m19.767166413s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896090823 (61h22m19.767173858s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896231440 (61h22m19.767314475s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896243242 (61h22m19.767326278s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896296854 (61h22m19.767379889s) [seg_pos={0 0}, upload=53840]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 04:51:19.896407245 (61h22m19.76749028s) [upload=53840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:51:19.896415951 (61h22m19.767498986s) [upload=53840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:54:58.203928113 (61h25m58.075011158s) [upload=53840]: Done uploading segments... Dec 8 04:54:58.203975553 (61h25m58.075058588s) [upload=53840]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.305844529s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [..........XX................................................] duration: 33.780366ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/2af55367efa5ea5e FAIL (2025-12-08 05:00:18 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 33.811474ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [...........XX...............................................] duration: 30.484078ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/2af55367efa5ea5e FAIL (2025-12-08 05:00:00 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 30.600991ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 698.165493ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4679b6ba9d140ac6 FAIL (2025-12-08 04:59:57 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 698.193006ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X......................X.] duration: 5.844517663s trace: https://jaeger.dev.storj.io/trace/67bd73b777c66f73 FAIL (2025-12-08 04:59:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.844531759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X..X..........X............] duration: 5.858469483s trace: https://jaeger.dev.storj.io/trace/327d8af62af2ad4a FAIL (2025-12-08 04:59:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 04:41:10.986572683 (61h12m10.857655728s) [upload=53698]: Starting upload Dec 8 04:41:10.986697500 (61h12m10.857780545s) [upload=53698]: Done starting upload Dec 8 04:41:10.986782503 (61h12m10.857865538s) [upload=53698]: Uploading segments... Dec 8 04:41:10.987063708 (61h12m10.858146753s) [upload=53698]: Got next segment. Inline: false Dec 8 04:41:10.987116498 (61h12m10.858199543s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 04:41:10.987121758 (61h12m10.858204803s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 04:41:17.462876699 (61h12m17.333959744s) [seg_pos={0 0}, upload=53698]: Begin upload segment... Dec 8 04:41:17.462933336 (61h12m17.334016381s) [seg_pos={0 0}, upload=53698]: Done begin upload segment. Dec 8 04:41:17.462963864 (61h12m17.334046909s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464241417 (61h12m17.335324452s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464758772 (61h12m17.335841817s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465245028 (61h12m17.336328063s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465727617 (61h12m17.336810672s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.466300738 (61h12m17.337383783s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.467904602 (61h12m17.338987647s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.469643584 (61h12m17.340726629s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.470720414 (61h12m17.341803449s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.472841653 (61h12m17.343924708s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.474783842 (61h12m17.345866877s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.478925869 (61h12m17.350008904s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.482146932 (61h12m17.353229977s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.484118316 (61h12m17.355201361s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.486216912 (61h12m17.357299957s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.487427918 (61h12m17.358510953s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:19.196950317 (61h12m19.068033352s) [upload=53698]: Next returned nil segment Dec 8 04:41:19.196972289 (61h12m19.068055324s) [upload=53698]: Waiting for error group managing segments... Dec 8 04:41:19.470957375 (61h12m19.34204042s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.478965694 (61h12m19.350048739s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.488459723 (61h12m19.359542758s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.491834840 (61h12m19.362917885s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.496467461 (61h12m19.367550506s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.497643140 (61h12m19.368726185s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.513620774 (61h12m19.384703809s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.519169180 (61h12m19.390252225s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.582513759 (61h12m19.453596794s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.600844513 (61h12m19.471927549s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.627512986 (61h12m19.498596031s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.923827261 (61h12m19.794910306s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940676124 (61h12m19.811759159s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940696483 (61h12m19.811779518s) [seg_pos={0 0}, upload=53698]: Segment reached optimal threshold of 13 pieces. Dec 8 04:41:19.940920650 (61h12m19.812003695s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.940927292 (61h12m19.812010347s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941137082 (61h12m19.812220117s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941146079 (61h12m19.812229124s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941154806 (61h12m19.812237851s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941162220 (61h12m19.812245255s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941203298 (61h12m19.812286343s) [seg_pos={0 0}, upload=53698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 04:41:19.941272510 (61h12m19.812355555s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:41:19.941277359 (61h12m19.812360404s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:50:23.376780170 (61h21m23.247863205s) [upload=53698]: Done uploading segments... Dec 8 04:50:23.376921268 (61h21m23.248004313s) [upload=53698]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.85849448s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X......................X] duration: 1m0.000886848s trace: https://jaeger.dev.storj.io/trace/67bd73b777c66f73 FAIL (2025-12-08 04:59:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000975938s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 471.576214ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/17143a9730f83ede FAIL (2025-12-08 04:59:41 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 471.607223ms core satellite eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.907871336s trace: https://jaeger.dev.storj.io/trace/6506fd1cc11bbd33 FAIL (2025-12-08 04:59:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.907898859s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X............X.X.........................................] duration: 3.440748107s trace: https://jaeger.dev.storj.io/trace/498ca12c4376e6b3 FAIL (2025-12-08 04:59:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440776231s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 694.324328ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4679b6ba9d140ac6 FAIL (2025-12-08 04:59:30 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 694.351381ms core satellite us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.260607641s trace: https://jaeger.dev.storj.io/trace/4acac71edcec4b82 FAIL (2025-12-08 04:59:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088365227 (61h22m16.959448272s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088855982 (61h22m16.959939017s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088942767 (61h22m16.960025813s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.089748512 (61h22m16.960831557s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.090080424 (61h22m16.961163469s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091013550 (61h22m16.962096595s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091792825 (61h22m16.96287587s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092319577 (61h22m16.963402613s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092355025 (61h22m16.96343806s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.095973455 (61h22m16.9670565s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.910383137 (61h22m17.781466182s) [upload=53840]: Next returned nil segment Dec 8 04:51:17.910461035 (61h22m17.781544071s) [upload=53840]: Waiting for error group managing segments... Dec 8 04:51:18.305376175 (61h22m18.176459221s) [noise=true , address=185.167.204.205:4030 , piece_id=ZHM2LCLRH2HGHUTQZLNHS5VN4SHQUHEXFLKSSG5VDO2O2U562TMA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.305953454 (61h22m18.177036499s) [noise=true , address=185.167.204.205:4025 , piece_id=4UDNENFCOE4JJ2EFKPSHEQQNX5FAWSH6DC3APZWRYSZWLCUCDCXA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307083516 (61h22m18.178166562s) [noise=true , address=185.167.204.205:4010 , piece_id=GTDYB6NWLDINXDPF7YDSXX3JPCEFX2CNVRX5567KDNMYQMZVWFCA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307746278 (61h22m18.178829313s) [noise=true , address=185.167.204.205:4006 , piece_id=QFJ66UCPW2MUX3OUZ64CQLIALWE6LRNOOFFVNMCAPXIXBLNGBQAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.308094291 (61h22m18.179177336s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.309637910 (61h22m18.180720945s) [noise=true , address=185.167.204.206:4043 , piece_id=EBRDIPWFGIK6BKYS5NWMWCQNRDTAO3AHDIFLRVKSR7HHJIS3PZBQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.316825457 (61h22m18.187908502s) [noise=true , address=185.167.204.206:4068 , piece_id=WOUIXUAMIW7OTF2YNM5SAAKWRZLEDAASLVRV3CIAQWRH7F7WKKDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.318376030 (61h22m18.189459075s) [noise=true , address=185.167.204.205:4013 , piece_id=WGR4SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.319751539 (61h22m18.190834574s) [noise=true , address=185.167.204.206:4050 , piece_id=TZO4WU4RMPPNA5ME525DJ34O3YBGLTRYFFWXRRDMPCRTO6LTFVDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.323889458 (61h22m18.194972503s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324447421 (61h22m18.195530466s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324906244 (61h22m18.195989289s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.886193526 (61h22m18.757276571s) [noise=true , address=81.171.14.70:4029 , piece_id=GEDBQ6CWAOJ67P6J6S7LD4EWZRYZV2A6N5V2F4A3HR66L6FERXNQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.896622956 (61h22m18.767706001s) [noise=true , address=81.171.14.70:4033 , piece_id=KVJN4PTIF46QANQMEX4WFBGN3DSOK3GU4B5VSX366ZXHLVNBAEAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.934515209 (61h22m18.805598244s) [noise=true , address=81.171.14.70:4045 , piece_id=J64JUTEWTUGOYDNW4PRLZYZZ24OSZSM6UZJQ7LGJLHTXCXRMSNZA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.940162311 (61h22m18.811245357s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.980483741 (61h22m18.851566776s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.308222254 (61h22m19.179305289s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.493269087 (61h22m19.364352132s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.546757438 (61h22m19.417840473s) [noise=true , address=89.149.201.147:4003 , piece_id=4LLUR2ZGUCDHESM7VRUFMOWDHOEOPJG7CVPOMOP2PUXH5Y6GMVUQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.561727113 (61h22m19.432810158s) [noise=true , address=89.149.201.147:4017 , piece_id=7KT2CZZXUVI6O5IO5KT2DN7VWYRMRMUDAQ4GVU4DZJWTHBE6RCIA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.574035961 (61h22m19.445118996s) [noise=true , address=89.149.201.147:4007 , piece_id=LZD4RYTCP2LPETVP5IHQSYPOS3ESASCFHSFWCFPXFOOUVAIJOAOA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.580254873 (61h22m19.451337918s) [noise=true , address=89.149.201.147:4011 , piece_id=AJRFRVNCSLZAHEVEDDQ5PRYPBVQDKZY6H7SR2HB7ZE4NMTG5AWEQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.619120290 (61h22m19.490203335s) [noise=true , address=89.149.201.147:4015 , piece_id=5QD6RF7RUQRCQTQFGLOKJKO6ULYJETBIC5NGRHB4WO3K7SSSELQQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.874085720 (61h22m19.745168765s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895800389 (61h22m19.766883434s) [noise=true , address=89.149.201.147:4020 , piece_id=QAYH5YBVG52KRKGV5GGDLAY2ZQGXBUL7V2IC2ZJYIALAFCSJRWCQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895827541 (61h22m19.766910596s) [seg_pos={0 0}, upload=53840]: Segment reached optimal threshold of 26 pieces. Dec 8 04:51:19.896083378 (61h22m19.767166413s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896090823 (61h22m19.767173858s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896231440 (61h22m19.767314475s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896243242 (61h22m19.767326278s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896296854 (61h22m19.767379889s) [seg_pos={0 0}, upload=53840]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 04:51:19.896407245 (61h22m19.76749028s) [upload=53840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:51:19.896415951 (61h22m19.767498986s) [upload=53840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:54:58.203928113 (61h25m58.075011158s) [upload=53840]: Done uploading segments... Dec 8 04:54:58.203975553 (61h25m58.075058588s) [upload=53840]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.260635404s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [............XX..............................................] duration: 31.030759ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/2af55367efa5ea5e FAIL (2025-12-08 04:59:18 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 31.058754ms edge linksharing bgp ap1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.215198799s trace: https://jaeger.dev.storj.io/trace/4acac71edcec4b82 FAIL (2025-12-08 04:59:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088365227 (61h22m16.959448272s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088855982 (61h22m16.959939017s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.088942767 (61h22m16.960025813s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.089748512 (61h22m16.960831557s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.090080424 (61h22m16.961163469s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091013550 (61h22m16.962096595s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.091792825 (61h22m16.96287587s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092319577 (61h22m16.963402613s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.092355025 (61h22m16.96343806s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.095973455 (61h22m16.9670565s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Uploading piece... Dec 8 04:51:17.910383137 (61h22m17.781466182s) [upload=53840]: Next returned nil segment Dec 8 04:51:17.910461035 (61h22m17.781544071s) [upload=53840]: Waiting for error group managing segments... Dec 8 04:51:18.305376175 (61h22m18.176459221s) [noise=true , address=185.167.204.205:4030 , piece_id=ZHM2LCLRH2HGHUTQZLNHS5VN4SHQUHEXFLKSSG5VDO2O2U562TMA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.305953454 (61h22m18.177036499s) [noise=true , address=185.167.204.205:4025 , piece_id=4UDNENFCOE4JJ2EFKPSHEQQNX5FAWSH6DC3APZWRYSZWLCUCDCXA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307083516 (61h22m18.178166562s) [noise=true , address=185.167.204.205:4010 , piece_id=GTDYB6NWLDINXDPF7YDSXX3JPCEFX2CNVRX5567KDNMYQMZVWFCA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.307746278 (61h22m18.178829313s) [noise=true , address=185.167.204.205:4006 , piece_id=QFJ66UCPW2MUX3OUZ64CQLIALWE6LRNOOFFVNMCAPXIXBLNGBQAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.308094291 (61h22m18.179177336s) [noise=true , address=185.167.204.205:4026 , piece_id=53HQRGSY5ZRJUF25YRHO5HNDHMXIJTHIWUZHKQ5Y5OEOBKHPEOJQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.309637910 (61h22m18.180720945s) [noise=true , address=185.167.204.206:4043 , piece_id=EBRDIPWFGIK6BKYS5NWMWCQNRDTAO3AHDIFLRVKSR7HHJIS3PZBQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.316825457 (61h22m18.187908502s) [noise=true , address=185.167.204.206:4068 , piece_id=WOUIXUAMIW7OTF2YNM5SAAKWRZLEDAASLVRV3CIAQWRH7F7WKKDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.318376030 (61h22m18.189459075s) [noise=true , address=185.167.204.205:4013 , piece_id=WGR4SQ4WX3ZVOFK7C7GHXC5EXQ3EL6R5SSPLGNGEPE3LW22B3X6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.319751539 (61h22m18.190834574s) [noise=true , address=185.167.204.206:4050 , piece_id=TZO4WU4RMPPNA5ME525DJ34O3YBGLTRYFFWXRRDMPCRTO6LTFVDA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.323889458 (61h22m18.194972503s) [noise=true , address=185.167.204.206:4053 , piece_id=WJQ4PGVZG4UIVZCNMK5M666KO7JB6NCLVTFCTKSDBPTBHVQVEIDQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324447421 (61h22m18.195530466s) [noise=true , address=185.167.204.206:4071 , piece_id=2HHZANMAZYOMJKJLVA5WTLCUDI26QKNZXK3DNMO7VW63OOQPQURA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.324906244 (61h22m18.195989289s) [noise=true , address=185.167.204.206:4066 , piece_id=SREKPZM2QEO6G45HBAEMQRNEX2HU24OESJYPZCXDTCOVGF6U2E6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.886193526 (61h22m18.757276571s) [noise=true , address=81.171.14.70:4029 , piece_id=GEDBQ6CWAOJ67P6J6S7LD4EWZRYZV2A6N5V2F4A3HR66L6FERXNQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.896622956 (61h22m18.767706001s) [noise=true , address=81.171.14.70:4033 , piece_id=KVJN4PTIF46QANQMEX4WFBGN3DSOK3GU4B5VSX366ZXHLVNBAEAQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.934515209 (61h22m18.805598244s) [noise=true , address=81.171.14.70:4045 , piece_id=J64JUTEWTUGOYDNW4PRLZYZZ24OSZSM6UZJQ7LGJLHTXCXRMSNZA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.940162311 (61h22m18.811245357s) [noise=true , address=81.171.14.70:4040 , piece_id=XEYMUEQXSOPJRZQYM2IEQYJRNVSWH4YDOXN4QL72B4GA66HEIHHQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:18.980483741 (61h22m18.851566776s) [noise=true , address=81.171.14.70:4025 , piece_id=A2TMMBZ2ZVUUCKWM3QQF7HT6TOER4QRFTOCWEK2F64HQUGG7BYIQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.308222254 (61h22m19.179305289s) [noise=true , address=81.171.14.70:4044 , piece_id=6E3SFEOBLBCVC7UDHAEEJYGDH4RBCMIEAIK776ATBZGOKOC5WWAA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.493269087 (61h22m19.364352132s) [noise=true , address=89.149.201.147:4010 , piece_id=UBHFHCVJMR3IDDBYIPDH4AT3AIOINGVXTBHIESGBFQ54B7WXFZPQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.546757438 (61h22m19.417840473s) [noise=true , address=89.149.201.147:4003 , piece_id=4LLUR2ZGUCDHESM7VRUFMOWDHOEOPJG7CVPOMOP2PUXH5Y6GMVUQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.561727113 (61h22m19.432810158s) [noise=true , address=89.149.201.147:4017 , piece_id=7KT2CZZXUVI6O5IO5KT2DN7VWYRMRMUDAQ4GVU4DZJWTHBE6RCIA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.574035961 (61h22m19.445118996s) [noise=true , address=89.149.201.147:4007 , piece_id=LZD4RYTCP2LPETVP5IHQSYPOS3ESASCFHSFWCFPXFOOUVAIJOAOA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.580254873 (61h22m19.451337918s) [noise=true , address=89.149.201.147:4011 , piece_id=AJRFRVNCSLZAHEVEDDQ5PRYPBVQDKZY6H7SR2HB7ZE4NMTG5AWEQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.619120290 (61h22m19.490203335s) [noise=true , address=89.149.201.147:4015 , piece_id=5QD6RF7RUQRCQTQFGLOKJKO6ULYJETBIC5NGRHB4WO3K7SSSELQQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.874085720 (61h22m19.745168765s) [noise=true , address=89.149.201.147:4021 , piece_id=DG5PEJPQMERSJUC6RMZYMSYIZLRC5Q7M346EQEGZZ7SL7Z6N4H7Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895800389 (61h22m19.766883434s) [noise=true , address=89.149.201.147:4020 , piece_id=QAYH5YBVG52KRKGV5GGDLAY2ZQGXBUL7V2IC2ZJYIALAFCSJRWCQ, seg_pos={0 0}, upload=53840]: Done uploading piece. err: <nil> Dec 8 04:51:19.895827541 (61h22m19.766910596s) [seg_pos={0 0}, upload=53840]: Segment reached optimal threshold of 26 pieces. Dec 8 04:51:19.896083378 (61h22m19.767166413s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896090823 (61h22m19.767173858s) [noise=true , address=89.149.201.147:4024 , piece_id=DYQMJPS4UHLFJLGIGL2J64QK4LMBPBIQSZAJ2TEDSF6UZMDKKQFA, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896231440 (61h22m19.767314475s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 8 04:51:19.896243242 (61h22m19.767326278s) [noise=true , address=89.149.201.147:4019 , piece_id=XHSINLNR3T7Y77HANA42AL7APB7L3YXP4PVVPNKR25OZCCHAFJ6Q, seg_pos={0 0}, upload=53840]: Long tail cancellation due to optimal threshold Dec 8 04:51:19.896296854 (61h22m19.767379889s) [seg_pos={0 0}, upload=53840]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 04:51:19.896407245 (61h22m19.76749028s) [upload=53840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:51:19.896415951 (61h22m19.767498986s) [upload=53840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:54:58.203928113 (61h25m58.075011158s) [upload=53840]: Done uploading segments... Dec 8 04:54:58.203975553 (61h25m58.075058588s) [upload=53840]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.215231843s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X..X..........X...........] duration: 5.758823709s trace: https://jaeger.dev.storj.io/trace/327d8af62af2ad4a FAIL (2025-12-08 04:59:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 04:41:10.986572683 (61h12m10.857655728s) [upload=53698]: Starting upload Dec 8 04:41:10.986697500 (61h12m10.857780545s) [upload=53698]: Done starting upload Dec 8 04:41:10.986782503 (61h12m10.857865538s) [upload=53698]: Uploading segments... Dec 8 04:41:10.987063708 (61h12m10.858146753s) [upload=53698]: Got next segment. Inline: false Dec 8 04:41:10.987116498 (61h12m10.858199543s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 04:41:10.987121758 (61h12m10.858204803s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 04:41:17.462876699 (61h12m17.333959744s) [seg_pos={0 0}, upload=53698]: Begin upload segment... Dec 8 04:41:17.462933336 (61h12m17.334016381s) [seg_pos={0 0}, upload=53698]: Done begin upload segment. Dec 8 04:41:17.462963864 (61h12m17.334046909s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464241417 (61h12m17.335324452s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.464758772 (61h12m17.335841817s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465245028 (61h12m17.336328063s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.465727617 (61h12m17.336810672s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.466300738 (61h12m17.337383783s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.467904602 (61h12m17.338987647s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.469643584 (61h12m17.340726629s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.470720414 (61h12m17.341803449s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.472841653 (61h12m17.343924708s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.474783842 (61h12m17.345866877s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.478925869 (61h12m17.350008904s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.482146932 (61h12m17.353229977s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.484118316 (61h12m17.355201361s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.486216912 (61h12m17.357299957s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:17.487427918 (61h12m17.358510953s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Uploading piece... Dec 8 04:41:19.196950317 (61h12m19.068033352s) [upload=53698]: Next returned nil segment Dec 8 04:41:19.196972289 (61h12m19.068055324s) [upload=53698]: Waiting for error group managing segments... Dec 8 04:41:19.470957375 (61h12m19.34204042s) [noise=true , address=145.79.227.40:4051 , piece_id=Z3JV2JYCGSZTSOKXPQFLG3LFXE3XSZXMEI6RKLWR7AIJWVE7F5VA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.478965694 (61h12m19.350048739s) [noise=true , address=145.79.227.22:4510 , piece_id=ME3CNTADT3MKZ7A7O35HQ77I3QEJ4RDHZ3HWNXTK74WCXXTVFGHA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.488459723 (61h12m19.359542758s) [noise=true , address=145.79.227.29:4943 , piece_id=W7RBOTEMANEW5K2RNN5VH25ZZWR7ENEZIWZYKGNL7CLYMFP5KSYA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.491834840 (61h12m19.362917885s) [noise=true , address=145.79.227.39:4014 , piece_id=42YQDT5VNDBCG42TGQE5XEO7V4M3FLXEBPCWNWF4VTDD4GVTD2MQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.496467461 (61h12m19.367550506s) [noise=true , address=145.79.227.28:4725 , piece_id=2KZFS66KORIJR7AJXPFK2YDIU5BGRNNYFUSMF2QCJHBXH4UJ5VLQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.497643140 (61h12m19.368726185s) [noise=true , address=145.79.227.26:4653 , piece_id=VXRBIXTQMICMWYO3QUTXC6WYEAMWLH5HDM6DGA3QSVJT6XUYQ4CQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.513620774 (61h12m19.384703809s) [noise=true , address=145.79.227.21:4474 , piece_id=ALCSXYN2JKG3WWKBYAGU3HSJGILRK324Q6BFUB3XAN7BRWIFUS6Q, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.519169180 (61h12m19.390252225s) [noise=true , address=145.79.227.24:4583 , piece_id=D66DYNF7HFKQXG2BZ7A7MS4TQQEUT7VXEUB4Q6K5E45QQ3KBLRPQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.582513759 (61h12m19.453596794s) [noise=true , address=77.90.31.18:4807 , piece_id=CG6IOIJXBGUWZHHSTI3CYHGQFHKJIUENOO7SXIMHMSKPT6KJC7OQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.600844513 (61h12m19.471927549s) [noise=true , address=77.90.31.13:4628 , piece_id=U3GVPVFRIIIOKMG2LTQZ7HLFQ4CKA3PJ7KDMTN37SRUVLXQ7PBHQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.627512986 (61h12m19.498596031s) [noise=true , address=77.90.31.10:4518 , piece_id=PVAIQII44G5U6ULAIESSKG65ZG4NH7DLS4DR2D4PRVULTRBDN2UQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.923827261 (61h12m19.794910306s) [noise=true , address=145.79.227.42:4123 , piece_id=SUGMMCANVZOSRM6KJF2JZFNGKCS76A3BIVR7FPEVGIFS3UXYSCBA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940676124 (61h12m19.811759159s) [noise=true , address=145.79.227.26:4654 , piece_id=KRO6XVQCHI3CYVUGBH4N5LLMUDTN2TTI3D2TR7F5YB24KOCIKDJQ, seg_pos={0 0}, upload=53698]: Done uploading piece. err: <nil> Dec 8 04:41:19.940696483 (61h12m19.811779518s) [seg_pos={0 0}, upload=53698]: Segment reached optimal threshold of 13 pieces. Dec 8 04:41:19.940920650 (61h12m19.812003695s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.940927292 (61h12m19.812010347s) [noise=true , address=145.79.227.33:4871 , piece_id=CSYTGISPEKL3YEHEE5YLAPJQJ76C7H4KR44ASGSWPHWIQCD4R7RA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941137082 (61h12m19.812220117s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941146079 (61h12m19.812229124s) [noise=true , address=145.79.227.21:4475 , piece_id=JZSLVHKHWCDZQM64QVSQYXZPORC63IY3U43X73Q5VPUDD3XCPKQA, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941154806 (61h12m19.812237851s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 8 04:41:19.941162220 (61h12m19.812245255s) [noise=true , address=145.79.227.27:4691 , piece_id=ESMSMYRQCCYAKGQYZCILQNNWHV3WACMYBDPEAEE44OUGZK3OGK3A, seg_pos={0 0}, upload=53698]: Long tail cancellation due to optimal threshold Dec 8 04:41:19.941203298 (61h12m19.812286343s) [seg_pos={0 0}, upload=53698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 04:41:19.941272510 (61h12m19.812355555s) [upload=53698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 04:41:19.941277359 (61h12m19.812360404s) [upload=53698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 04:50:23.376780170 (61h21m23.247863205s) [upload=53698]: Done uploading segments... Dec 8 04:50:23.376921268 (61h21m23.248004313s) [upload=53698]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.758848535s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 693.108804ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4679b6ba9d140ac6 FAIL (2025-12-08 04:59:06 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 693.162115ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X............X.X........................................] duration: 3.541892805s trace: https://jaeger.dev.storj.io/trace/498ca12c4376e6b3 FAIL (2025-12-08 04:59:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.541920797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.597049564s trace: https://jaeger.dev.storj.io/trace/6506fd1cc11bbd33 FAIL (2025-12-08 04:58:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.597076407s edge gateway us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 477.557434ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/17143a9730f83ede FAIL (2025-12-08 04:58:51 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 477.60312ms core satellite eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 695.604857ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4679b6ba9d140ac6 FAIL (2025-12-08 04:58:49 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 695.63816ms core satellite us1

Orbiter Check History

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)