Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 839688 reports (Page 1845 of 16794)

checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 637.445318ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:56:29 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 637.466187ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X..X........................] duration: 5.736700043s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:56:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.736727024s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.867823903s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:56:05 UTC) orbiter warning: context deadline exceeded Duration: 6.867851155s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.706215062s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:55:49 UTC) orbiter warning: context deadline exceeded Duration: 6.706239219s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X.....X..............................................] duration: 3.825576928s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:55:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.825637916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X.....X.............................................] duration: 3.902706639s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:55:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.902746616s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 636.121039ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:55:20 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 636.145414ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X..X.......................] duration: 5.730876857s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:55:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.730905242s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X.....X............................................] duration: 3.825057486s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:54:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.825258771s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.806877663s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:54:54 UTC) orbiter warning: context deadline exceeded Duration: 6.806902161s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 635.417263ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:54:29 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 635.442893ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X.....X...........................................] duration: 3.941087008s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:54:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.941111586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X..X......................] duration: 6.600068184s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:54:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 6.60009194s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.774348395s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:54:07 UTC) orbiter warning: context deadline exceeded Duration: 6.774373493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X.....X..........................................] duration: 3.863908128s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:53:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.863923398s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 633.733172ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:53:44 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 633.766715ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.629173615s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:53:15 UTC) orbiter warning: context deadline exceeded Duration: 6.62922724s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X.....X.........................................] duration: 3.730354957s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:53:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.730379314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X..X.....................] duration: 5.72375616s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:52:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.723778182s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.624375538s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:52:59 UTC) orbiter warning: context deadline exceeded Duration: 6.624403393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.599141019s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:52:44 UTC) orbiter warning: context deadline exceeded Duration: 6.599160706s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X.....X........................................] duration: 4.742131005s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:52:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.742153709s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 639.30833ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:52:34 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 639.336273ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..X....................] duration: 5.690226069s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:52:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.690250005s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X.....X.......................................] duration: 3.802369767s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:52:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.80239723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..X...................] duration: 5.648268426s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:52:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.648294145s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.671668489s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:52:11 UTC) orbiter warning: context deadline exceeded Duration: 6.671692376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.798570906s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:51:46 UTC) orbiter warning: context deadline exceeded Duration: 6.79859461s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 637.137688ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:51:45 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 637.162575ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X.....X......................................] duration: 3.772051025s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:51:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.772073888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..X..................] duration: 5.712990277s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:51:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.713015724s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X.....X.....................................] duration: 3.745653913s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:50:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.745671987s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.620680499s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:50:44 UTC) orbiter warning: context deadline exceeded Duration: 6.620747206s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 633.837521ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:50:39 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 633.859823ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..X.................] duration: 5.766179669s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:50:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.766217932s core uplink us-select-1
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 647.536423ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:50:04 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 647.563614ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..X................] duration: 5.776088576s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:49:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.776108634s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X.....X....................................] duration: 3.724532258s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:49:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.724551224s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 650.79991ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:49:41 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 650.818636ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.582033869s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:49:30 UTC) orbiter warning: context deadline exceeded Duration: 6.582078806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X..X...............] duration: 5.700581607s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:48:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.700606203s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X.....X...................................] duration: 3.667363179s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:48:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.667388538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..X..............] duration: 5.768684225s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:48:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.768710013s core uplink us-select-1
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 650.569288ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:48:33 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 650.599778ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X..X.............] duration: 5.764046378s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:48:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.764068099s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.773096637s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:48:20 UTC) orbiter warning: context deadline exceeded Duration: 6.773114993s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X.....X..................................] duration: 3.947066689s trace: https://jaeger.dev.storj.io/trace/7f202defd7407eb4 FAIL (2026-05-14 04:48:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.947085824s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 654.20628ms remote: "tcp:35.209.24.65:443" target: https://version.storj.io/ trace: https://jaeger.dev.storj.io/trace/61406304dd190e9e FAIL (2026-05-14 04:48:05 UTC) orbiter warning: Get "https://version.storj.io/": read tcp 188.166.225.170:46460->35.209.24.65:443: read: connection reset by peer Duration: 654.225267ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.864695081s trace: https://jaeger.dev.storj.io/trace/6cb443d3928d2110 FAIL (2026-05-14 04:48:00 UTC) orbiter warning: context deadline exceeded Duration: 6.864718627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X..X............] duration: 5.698597319s trace: https://jaeger.dev.storj.io/trace/793bc7a1d93181c2 FAIL (2026-05-14 04:47:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== in May 14 04:37:54.692610915 (321h50m5.21343466s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 14 04:37:54.930969401 (321h50m5.451793147s) [seg_pos={0 0}, upload=308354]: Begin upload segment... May 14 04:37:54.931049184 (321h50m5.451872979s) [seg_pos={0 0}, upload=308354]: Done begin upload segment. May 14 04:37:54.931156629 (321h50m5.451980374s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931187608 (321h50m5.452011383s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931318638 (321h50m5.452142383s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.931998635 (321h50m5.45282238s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932196884 (321h50m5.453020669s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932509510 (321h50m5.453333255s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.932689223 (321h50m5.453512968s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933251857 (321h50m5.454075612s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.933832485 (321h50m5.45465623s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934174568 (321h50m5.454998313s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934314514 (321h50m5.455138259s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934671185 (321h50m5.45549494s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.934686375 (321h50m5.45551013s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935183793 (321h50m5.456007538s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.935759361 (321h50m5.456583136s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:54.936232875 (321h50m5.45705662s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Uploading piece... May 14 04:37:56.659124982 (321h50m7.179948717s) [upload=308354]: Next returned nil segment May 14 04:37:56.659141724 (321h50m7.179965459s) [upload=308354]: Waiting for error group managing segments... May 14 04:37:56.945082708 (321h50m7.465906453s) [noise=true , address=145.79.227.25:4618 , piece_id=E2QGMZATAZ6GS7DPQMPAJC7TUADR7CDCCJIYY234WGW52USXV5SA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960510078 (321h50m7.481333823s) [noise=true , address=145.79.227.37:4566 , piece_id=RSMGGNYFY5VIWNS6FLHFCVKVOPTQP5QUL7E5LRG6AKWDVF2I3XAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.960815721 (321h50m7.481639466s) [noise=true , address=145.79.227.45:4230 , piece_id=L3UN5CZ4ZGTUOAPG6CQY4BNYHI2EEE65LRGARQXYIOOIABW4D2FQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.963554487 (321h50m7.484378232s) [noise=true , address=145.79.227.4:4190 , piece_id=Z2KPHHRAEFXQ54GO735E7RUJ64MERQY2NTHTWMXL6TYC4YG2W2IQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964875267 (321h50m7.485699012s) [noise=true , address=145.79.227.10:4405 , piece_id=3EBKQQDFUNITALD5I4QMTOL66EEMKNON5PQE3IRZZFTSJF5NWZ5Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.964957885 (321h50m7.48578163s) [noise=true , address=145.79.227.8:4332 , piece_id=QDDEHIDSKQQX7N4LVSCWGE4TUVCTYTH5T5KMMI547PJCZ7WMLBFA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.973697743 (321h50m7.494521478s) [noise=true , address=145.79.227.23:4546 , piece_id=3RJL2JPERMTZ5AGIEHJ2M5FXBDTSC6I4BNWLAWJSZQ5LUW7IY5HA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.976134162 (321h50m7.496957907s) [noise=true , address=145.79.227.21:4477 , piece_id=5MNJ3FKZH45LM6NJEWOMUVCAYXD44G4DMPBN53M5SY7KDO6FKZYQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:56.978984722 (321h50m7.499808457s) [noise=true , address=145.79.227.39:4013 , piece_id=O6XTGBJWAOB2537CRM3EQQSQUOAIJDP6ARCXEVIGHD7GI6ROOSAA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.017257796 (321h50m7.538081541s) [noise=true , address=145.79.227.48:4306 , piece_id=PXQCVJDKUWZTAYPON6DXBOEMV2GLQOD76HXZWZHWZEABFIMFAM7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.024480008 (321h50m7.545303753s) [noise=true , address=145.79.227.12:4771 , piece_id=6QYCXP4TRPQ3Q2UXFHOL5HA22ZAJWHLZYN33ILMHO4GHWTVHAQLA, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.027084668 (321h50m7.547908413s) [noise=true , address=145.79.227.51:4412 , piece_id=3AWOXCPOPTF6C3W4IVZTF6SAP44UUBESVOJHP457FJ66EJII7QZQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044769395 (321h50m7.56559314s) [noise=true , address=145.79.227.40:4054 , piece_id=K6JH7YWWHNRUR67J4JABEKROHEM56PM2VEXQWGKVPXBIHA3VBG4A, seg_pos={0 0}, upload=308354]: Done uploading piece. err: <nil> May 14 04:37:57.044787289 (321h50m7.565611024s) [seg_pos={0 0}, upload=308354]: Satisfied cohort requirements after 13 pieces May 14 04:37:57.044949048 (321h50m7.565772793s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tdXgFPYY7uaVB2rqi5pTctyZpmXQcCpyMTkHy9q1r3H2T1BZN): protocol: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.8:4080: use of closed network connection May 14 04:37:57.044953506 (321h50m7.565777251s) [noise=true , address=45.91.116.8:4080 , piece_id=5THBVBHXGFAC4PR6EH2RKERCO4U7JA2IKAMGWGFICDQLE372XSVQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045112039 (321h50m7.565935774s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37446->45.91.116.5:4297: use of closed network connection May 14 04:37:57.045119323 (321h50m7.565943068s) [noise=true , address=45.91.116.5:4297 , piece_id=SART3ZNQLIYXFX2Y5IZVAQMEVMHK4MXBXPXZT2X7D7TBWMA4OXOQ, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045126506 (321h50m7.565950251s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41754->45.91.116.10:4082: use of closed network connection May 14 04:37:57.045133800 (321h50m7.565957545s) [noise=true , address=45.91.116.10:4082 , piece_id=HSVVFNYPI52SM3QRPBKTKRTQY6W3PGK2IMA46FUSQETPXWP4XH7Q, seg_pos={0 0}, upload=308354]: Long tail cancellation due to optimal threshold May 14 04:37:57.045209124 (321h50m7.566032869s) [seg_pos={0 0}, upload=308354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 14 04:37:57.045309666 (321h50m7.566133411s) [upload=308354]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 14 04:37:57.045314124 (321h50m7.566137869s) [upload=308354]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 14 04:38:01.690068805 (321h50m12.21089255s) [upload=308354]: Done uploading segments... May 14 04:38:01.690154709 (321h50m12.210978454s) [upload=308354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.698626746s core uplink us-select-1

1
...
1843
1844
1845
1846
1847
...
16794