Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 865121 reports (Page 11685 of 17303)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............XX...................................X.........] duration: 5.764663767s trace: https://jaeger.dev.storj.io/trace/3d538152cdb836c6 FAIL (2025-12-08 19:34:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.764693282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 7.114825392s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:33:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.114867523s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.70721324s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:33:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.707259068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.986484013s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:33:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.986510633s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.166983221s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:33:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.167013219s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X.....................................X......] duration: 3.446457072s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:33:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.446484143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.987611818s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:33:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.987640805s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X.............XX...................................X........] duration: 5.881357263s trace: https://jaeger.dev.storj.io/trace/3d538152cdb836c6 FAIL (2025-12-08 19:32:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.881386188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.67802147s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:32:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.678047871s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X.....................................X.....] duration: 3.472424445s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:32:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472479541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.99949291s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:32:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.999529721s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.68042742s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:32:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.680455984s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000456578s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:32:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000493879s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X.............XX...................................X.......] duration: 5.866187266s trace: https://jaeger.dev.storj.io/trace/3d538152cdb836c6 FAIL (2025-12-08 19:32:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.866218937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.602658705s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:32:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.602690566s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.68927024s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:32:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.689300699s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X.....................................X....] duration: 3.619903806s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:31:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.619934655s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X.............XX...................................X......] duration: 6.512771966s trace: https://jaeger.dev.storj.io/trace/3d538152cdb836c6 FAIL (2025-12-08 19:31:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.512802194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.021131775s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:31:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.021163696s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X.....................................X...] duration: 3.539843327s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:31:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539865839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.77115557s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:31:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.771185288s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X.............XX...................................X.....] duration: 5.546853205s trace: https://jaeger.dev.storj.io/trace/3d538152cdb836c6 FAIL (2025-12-08 19:31:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.547030081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.908068395s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:30:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.908091159s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.629069757s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:30:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.629101437s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.912103799s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:30:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.912133435s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X.....................................X..] duration: 3.635300973s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:30:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.635328875s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X.............XX...................................X....] duration: 5.79725247s trace: https://jaeger.dev.storj.io/trace/3d538152cdb836c6 FAIL (2025-12-08 19:29:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.797272248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X.....................................X.] duration: 3.430480867s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:29:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430542043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X.....................................X] duration: 5.000763452s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:29:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000785203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.963850029s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:29:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.963883763s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.82347876s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:29:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.823511201s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X.............XX...................................X...] duration: 5.628406991s trace: https://jaeger.dev.storj.io/trace/3d538152cdb836c6 FAIL (2025-12-08 19:28:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.628433169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.38840575s trace: https://jaeger.dev.storj.io/trace/25838e89bb595afa FAIL (2025-12-08 19:28:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.388430847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.112641473s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:28:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.112668313s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.708471214s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:28:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.708495762s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X.............XX...................................X..] duration: 5.820406862s trace: https://jaeger.dev.storj.io/trace/3d538152cdb836c6 FAIL (2025-12-08 19:27:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.820427961s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.512261493s trace: https://jaeger.dev.storj.io/trace/25838e89bb595afa FAIL (2025-12-08 19:27:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.512292864s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.890096263s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:27:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.890120169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.733298467s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:27:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.733334246s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X.............XX...................................X.] duration: 5.797997157s trace: https://jaeger.dev.storj.io/trace/3d538152cdb836c6 FAIL (2025-12-08 19:27:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.798014701s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X.......X.............XX...................................X] duration: 1m0.001555132s trace: https://jaeger.dev.storj.io/trace/3d538152cdb836c6 FAIL (2025-12-08 19:27:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001591381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.574568999s trace: https://jaeger.dev.storj.io/trace/25838e89bb595afa FAIL (2025-12-08 19:26:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.574592083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.979271822s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:26:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.979305557s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.687403346s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:26:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.68742077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.66697292s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:26:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.666999291s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 4.3027487s trace: https://jaeger.dev.storj.io/trace/25838e89bb595afa FAIL (2025-12-08 19:26:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.30277002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.866992947s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:26:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.867024606s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.758895094s trace: https://jaeger.dev.storj.io/trace/25838e89bb595afa FAIL (2025-12-08 19:26:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.758930189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 8.231906307s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:25:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 8.231935891s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.49366767s trace: https://jaeger.dev.storj.io/trace/25838e89bb595afa FAIL (2025-12-08 19:25:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.493693017s core uplink us-select-1 us1-placement-20

1
...
11683
11684
11685
11686
11687
...
17303