Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 341296 reports (Page 154 of 6826)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....X..XX....X...X.........X..XX.X....XX.X.....XX..........] duration: 5.649744647s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:32:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 5.649774032s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.X................................................] duration: 25.031096262s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:32:08 UTC) orbiter warning: context deadline exceeded Duration: 25.031133033s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......X..XX....X...X.........X..XX.X....XX.X.....XX.........] duration: 5.818977952s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:32:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 5.819009291s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.] duration: 3.553859794s trace: https://jaeger.dev.storj.io/trace/5941db0c2739fdf FAIL (2025-12-28 12:31:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.553875453s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [X.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX] duration: 5.001212108s trace: https://jaeger.dev.storj.io/trace/5941db0c2739fdf FAIL (2025-12-28 12:31:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001235943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 48.997727737s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:31:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 48.997753235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.X...............................................] duration: 18.848466855s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:31:20 UTC) orbiter warning: context deadline exceeded Duration: 18.848499609s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......X..XX....X...X.........X..XX.X....XX.X.....XX........] duration: 5.836997816s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:31:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 5.837025089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XX] duration: 5.001576131s trace: https://jaeger.dev.storj.io/trace/5c5f579ebc32991b FAIL (2025-12-28 12:30:47 UTC) orbiter warning: context deadline exceeded Duration: 5.001598584s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..X] duration: 5.000705183s trace: https://jaeger.dev.storj.io/trace/33aeae632669dd75 FAIL (2025-12-28 12:30:24 UTC) orbiter warning: context deadline exceeded Duration: 5.000726724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 44.575687824s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:30:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 44.575712981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.......X..XX....X...X.........X..XX.X....XX.X.....XX.......] duration: 5.975929538s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:30:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 5.975959955s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.X..............................................] duration: 7.669594613s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:30:16 UTC) orbiter warning: context deadline exceeded Duration: 7.669623478s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.X.............................................] duration: 16.065996974s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:29:54 UTC) orbiter warning: context deadline exceeded Duration: 16.06602561s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 43.460115861s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:29:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 43.460128344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [..XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..] duration: 3.521540341s trace: https://jaeger.dev.storj.io/trace/34654b3823b87bf8 FAIL (2025-12-28 12:29:27 UTC) orbiter warning: context deadline exceeded Duration: 3.521573395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.......X..XX....X...X.........X..XX.X....XX.X.....XX......] duration: 27.095404988s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:29:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 27.095423624s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.X............................................] duration: 15.269236023s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:29:17 UTC) orbiter warning: context deadline exceeded Duration: 15.269271411s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 16.314554835s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:28:52 UTC) orbiter warning: context deadline exceeded Duration: 16.31458883s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.004672455s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:28:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 1m0.00468015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [...XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX.] duration: 5.001522197s trace: https://jaeger.dev.storj.io/trace/34654b3823b87bf8 FAIL (2025-12-28 12:28:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001543268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX] duration: 5.001419084s trace: https://jaeger.dev.storj.io/trace/34654b3823b87bf8 FAIL (2025-12-28 12:28:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001434914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX.......X..XX....X...X.........X..XX.X....XX.X.....XX.....] duration: 8.216569887s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:28:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 8.216594745s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 6.571070255s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:28:06 UTC) orbiter warning: context deadline exceeded Duration: 6.571098459s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XXX.......X..XX....X...X.........X..XX.X....XX.X.....XX....] duration: 5.908289707s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:27:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 5.908318422s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.X] duration: 5.001547746s trace: https://jaeger.dev.storj.io/trace/6c52b04226fcb628 FAIL (2025-12-28 12:27:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001580458s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [.X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.] duration: 3.693065202s trace: https://jaeger.dev.storj.io/trace/5db446a04f41df16 FAIL (2025-12-28 12:27:11 UTC) orbiter warning: context deadline exceeded Duration: 3.693086514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 16.328193656s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:27:00 UTC) orbiter warning: context deadline exceeded Duration: 16.328212541s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XXX.......X..XX....X...X.........X..XX.X....XX.X.....XX...] duration: 5.90336981s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:26:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 5.903399356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...XXX.......X..XX....X...X.........X..XX.X....XX.X.....XX..] duration: 5.831143299s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:26:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 5.831176733s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX] duration: 5.000698801s trace: https://jaeger.dev.storj.io/trace/5db446a04f41df16 FAIL (2025-12-28 12:26:30 UTC) orbiter warning: context deadline exceeded Duration: 5.000721874s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 15.547063682s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:26:23 UTC) orbiter warning: context deadline exceeded Duration: 15.547081015s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [X..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.X] duration: 5.001513972s trace: https://jaeger.dev.storj.io/trace/7d60961bb5f351ff FAIL (2025-12-28 12:25:46 UTC) orbiter warning: context deadline exceeded Duration: 5.001542478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...XXX.......X..XX....X...X.........X..XX.X....XX.X.....XX.] duration: 5.894206149s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:25:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 5.894236399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX...XXX.......X..XX....X...X.........X..XX.X....XX.X.....XX] duration: 30.003516865s trace: https://jaeger.dev.storj.io/trace/3d3e62cac83d24dd FAIL (2025-12-28 12:25:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ) context deadline exceeded ======LOGS====== Dec 28 12:25:06.090297703 (548h56m5.961380738s) [upload=481320]: Starting upload Dec 28 12:25:06.090434744 (548h56m5.961517789s) [upload=481320]: Done starting upload Dec 28 12:25:06.090508043 (548h56m5.961591078s) [upload=481320]: Uploading segments... Dec 28 12:25:06.090635586 (548h56m5.961718631s) [upload=481320]: Got next segment. Inline: false Dec 28 12:25:06.090672076 (548h56m5.961755121s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:25:06.090677376 (548h56m5.961760421s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:25:06.335719768 (548h56m6.206802813s) [seg_pos={0 0}, upload=481320]: Begin upload segment... Dec 28 12:25:06.335773410 (548h56m6.206856455s) [seg_pos={0 0}, upload=481320]: Done begin upload segment. Dec 28 12:25:06.335793499 (548h56m6.206876544s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335810541 (548h56m6.206893616s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.335878089 (548h56m6.206961124s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336254937 (548h56m6.207337972s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336432335 (548h56m6.20751537s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336593973 (548h56m6.207677008s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.336760700 (548h56m6.207843745s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337071101 (548h56m6.208154136s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337346235 (548h56m6.20842927s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337627009 (548h56m6.208710044s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.337978348 (548h56m6.209061383s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338349065 (548h56m6.2094321s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.338512476 (548h56m6.209595521s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339012237 (548h56m6.210095282s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339458828 (548h56m6.210541873s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:06.339778597 (548h56m6.210861632s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Uploading piece... Dec 28 12:25:08.051151552 (548h56m7.922234597s) [upload=481320]: Next returned nil segment Dec 28 12:25:08.051169746 (548h56m7.922252791s) [upload=481320]: Waiting for error group managing segments... Dec 28 12:25:08.332000676 (548h56m8.203083711s) [noise=true , address=145.79.227.46:4602 , piece_id=EHCFT3ZTCFM6AJXRCUKODKSOD5CPLO4JFEVSVJUXX7CQX27LQYKQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.341435031 (548h56m8.212518086s) [noise=true , address=145.79.227.6:4258 , piece_id=WFB7TWSFRUYP7KTQJH3AIBK6A3EO4YSDG72427JF53HJ46FB6LTQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.346753948 (548h56m8.217836993s) [noise=true , address=145.79.227.52:4446 , piece_id=IBKOVJ5JG2L5OPYWY4ZGXJST5EZZV6UN6QFDDJRNGHEFFR5C75MA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.347554202 (548h56m8.218637247s) [noise=true , address=145.79.227.37:4565 , piece_id=RK5GVTJITCB5JDIZOCVWTAXRFU4BDTBQ53YZCLTF32MEJFKNVEJQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.348812860 (548h56m8.219895895s) [noise=true , address=145.79.227.27:4691 , piece_id=IDGZTASSLDPTZT2X334U4ZNCRX4KXUAUHJRH4VGIP4PU2CO4IFZA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.359506171 (548h56m8.230589216s) [noise=true , address=145.79.227.16:4911 , piece_id=5FBOGCEUNFOXX7PHHJPPRUWYQN6ZDK27BXYLN5FZYBNYUFX5OJ2Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.360449658 (548h56m8.231532693s) [noise=true , address=145.79.227.25:4619 , piece_id=DXFD2FRYP5JAKND7HJAAVSY3JFZLM6UDEQQQLNB4SE3BGGH55XUA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.361942270 (548h56m8.233025315s) [noise=true , address=145.79.227.32:4836 , piece_id=MV4IKU4SVNEGYGEW5VJHP4NACEZ7EQUVQYZHWWNTAOXZIMFDHJEQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.367036230 (548h56m8.238119275s) [noise=true , address=145.79.227.13:4803 , piece_id=CNG77GDEQYAMGX3N7OYSAVCLXFUQ33D7VVSHV7DXX2QERID2PYWQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.370562153 (548h56m8.241645198s) [noise=true , address=145.79.227.47:4265 , piece_id=SPBDZFPD5YJCYPJ4VSJKDDKSHO2GUWHOVAMS2HNFKCPO3OMI5CLQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.606570614 (548h56m8.477653649s) [noise=true , address=77.90.31.11:4556 , piece_id=RA3RH7UTRUWAV7B5IQDGEDTFLZMPMWAWJGACAYQSOJFPIBDG3ANQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.629341544 (548h56m8.500424589s) [noise=true , address=77.90.31.12:4590 , piece_id=KV3Y3JO2AR34EFULUYZMS43CONZEWHMGGVFGGZAN5S2ZQ7RW4OBA, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630537771 (548h56m8.501620807s) [noise=true , address=77.90.31.14:4664 , piece_id=LV3PEWT7YEI7BXP6LN7VFLH35GXL6WXY2BDSEGF5IALCFD52ZBHQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: <nil> Dec 28 12:25:08.630548151 (548h56m8.501631196s) [seg_pos={0 0}, upload=481320]: Segment reached optimal threshold of 13 pieces. Dec 28 12:25:08.630736019 (548h56m8.501819064s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630743523 (548h56m8.501826558s) [noise=true , address=45.91.116.5:4300 , piece_id=NUSAWWACECQFW643WIYKGOR6PCCLGO7ACSQKOMGZIAZDVBD3L3VQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.630976567 (548h56m8.502059612s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.630983180 (548h56m8.502066215s) [noise=true , address=77.90.31.19:4842 , piece_id=ORD5ACEOVDQPT4BKSRMFANPI4HB6HFOHVHWQTCKRXPVRFDTMUO7Q, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631099611 (548h56m8.502182656s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 12:25:08.631104991 (548h56m8.502188036s) [noise=true , address=45.91.116.10:4081 , piece_id=JHORDRH2IZ3QL65DTQ5KCMWPRSJBVYP4FOL5HV4IP3MSPEMCBEZQ, seg_pos={0 0}, upload=481320]: Long tail cancellation due to optimal threshold Dec 28 12:25:08.631120411 (548h56m8.502203456s) [seg_pos={0 0}, upload=481320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:25:08.631191316 (548h56m8.502274351s) [upload=481320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:25:08.631196095 (548h56m8.50227913s) [upload=481320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:25:08.926268503 (548h56m8.797351579s) [upload=481320]: Done uploading segments... Dec 28 12:25:08.926302889 (548h56m8.797385934s) [upload=481320]: Upload finished. err: <nil> Duration: 30.003548294s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 6.829403907s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:25:11 UTC) orbiter warning: context deadline exceeded Duration: 6.829437129s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.] duration: 5.001097539s trace: https://jaeger.dev.storj.io/trace/3b5ae53663861f01 FAIL (2025-12-28 12:24:33 UTC) orbiter warning: context deadline exceeded Duration: 5.001123839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX...XXX.......X..XX....X...X.........X..XX.X....XX.X.....X] duration: 30.002816867s trace: https://jaeger.dev.storj.io/trace/52f77a635ae351c7 FAIL (2025-12-28 12:24:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: 4GGISQWLDH7M7YZZIJHY3X5PXYFZDCNJSS4JGYMEDQDOXNLURP6A) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UY6CQUAWGN2TQWNB3PB6MRLEQ7WWD7O3C3DEIT5LCYTRSE6BZVJQ) context deadline exceeded ======LOGS====== Dec 28 12:23:53.758675969 (548h54m53.629759014s) [upload=481306]: Starting upload Dec 28 12:23:53.758787241 (548h54m53.629870296s) [upload=481306]: Done starting upload Dec 28 12:23:53.758862945 (548h54m53.62994598s) [upload=481306]: Uploading segments... Dec 28 12:23:53.759024924 (548h54m53.630107969s) [upload=481306]: Got next segment. Inline: false Dec 28 12:23:53.759101770 (548h54m53.630184815s) [upload=481306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:23:53.759107380 (548h54m53.630190425s) [upload=481306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:23:54.000955540 (548h54m53.872038606s) [seg_pos={0 0}, upload=481306]: Begin upload segment... Dec 28 12:23:54.001012539 (548h54m53.872095594s) [seg_pos={0 0}, upload=481306]: Done begin upload segment. Dec 28 12:23:54.001053858 (548h54m53.872136913s) [noise=true , address=45.91.116.9:4263 , piece_id=WEYPS25VOPTMTSEJL6VMLWIV6R36MVOKYEFZMGQJCXCRABUIZGQQ, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.001070719 (548h54m53.872153765s) [noise=true , address=45.91.116.12:4228 , piece_id=6K2V43NFIGDFCPIYC2NRYNCWOYGPABDE5IJ427HQLGS3ZR43M2JA, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.001192352 (548h54m53.872275397s) [noise=true , address=77.90.31.11:4553 , piece_id=4GGISQWLDH7M7YZZIJHY3X5PXYFZDCNJSS4JGYMEDQDOXNLURP6A, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.001932260 (548h54m53.873015345s) [noise=true , address=145.79.227.39:4015 , piece_id=ZGICJN6ZHBJVKIGF5JRAPK7K4LIMGVRRBQUAC5TL23DA2AVUZMAQ, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.002603478 (548h54m53.873686523s) [noise=true , address=145.79.227.9:4367 , piece_id=XTLQ3Q6J6URPFGQ33SR2L6BTQF4PCTNJBGDAPBFJ2ANEYNG7CO5Q, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.003087630 (548h54m53.874170675s) [noise=true , address=145.79.227.14:4838 , piece_id=U2V63ZXECWLPXPYBW3ZN347SABGUZBXHTSFQ5QT4XRDTXUKMIL3Q, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.003915777 (548h54m53.874998822s) [noise=true , address=145.79.227.42:4122 , piece_id=D6XH4GC7QKJNWS4H2XVNZHHAV4E3GYMQTWENMIPVGEFYWWDRNEWA, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.004446237 (548h54m53.875529282s) [noise=true , address=77.90.31.19:4844 , piece_id=UY6CQUAWGN2TQWNB3PB6MRLEQ7WWD7O3C3DEIT5LCYTRSE6BZVJQ, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.005101144 (548h54m53.876184189s) [noise=true , address=145.79.227.41:4087 , piece_id=33NLJDXQOAI3W4IGT2WH73LE4STU52KKREBSOZJMDO3A4KVQJ4VA, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.007252260 (548h54m53.878335305s) [noise=true , address=145.79.227.20:4439 , piece_id=OEQOKZUV6PF5PVCMB6K7O7E2ADGKLF7YYJXNETJAOL7JIL7LQYSA, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.008282141 (548h54m53.879365186s) [noise=true , address=145.79.227.2:4114 , piece_id=2F4ST2WYZVQQ4CDE57K47D3XMPJNO3YJNBLPPJAOQFYYRQAVWUDQ, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.010796218 (548h54m53.881879263s) [noise=true , address=145.79.227.16:4911 , piece_id=U5MAVG64WWGY7VXKJHJ6XVWO7DTKR37VMCLHDL4HKOSPSPGMYLKA, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.012845571 (548h54m53.883928616s) [noise=true , address=145.79.227.36:4519 , piece_id=MXI7X5SKSVRVYRAJWQQSR2JXEGKOT664G5UV36BJ36PORUNVLVSQ, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.014789582 (548h54m53.885872618s) [noise=true , address=77.90.31.10:4517 , piece_id=FVJMTGTEL6OCPGX3EQ62FCTXDZXNEZTXNOCA3OHREPO3SODUM6DQ, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.016598577 (548h54m53.887681622s) [noise=true , address=145.79.227.50:4374 , piece_id=Q6VGBXXCDR53FPI25VJMXGG4YKBJPYXMMAACNDP6VOTEKQZ43QKA, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:54.018462697 (548h54m53.889545742s) [noise=true , address=145.79.227.7:4294 , piece_id=K54RV6XXNZUTCG343DZLRVBH3SB45URWFU6ZTHRWR3ZQGQV2RM7Q, seg_pos={0 0}, upload=481306]: Uploading piece... Dec 28 12:23:55.615353849 (548h54m55.486436884s) [upload=481306]: Next returned nil segment Dec 28 12:23:55.615367726 (548h54m55.486450771s) [upload=481306]: Waiting for error group managing segments... Dec 28 12:23:56.007293689 (548h54m55.878376734s) [noise=true , address=145.79.227.14:4838 , piece_id=U2V63ZXECWLPXPYBW3ZN347SABGUZBXHTSFQ5QT4XRDTXUKMIL3Q, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.007845590 (548h54m55.878928645s) [noise=true , address=145.79.227.42:4122 , piece_id=D6XH4GC7QKJNWS4H2XVNZHHAV4E3GYMQTWENMIPVGEFYWWDRNEWA, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.013084126 (548h54m55.884167171s) [noise=true , address=145.79.227.41:4087 , piece_id=33NLJDXQOAI3W4IGT2WH73LE4STU52KKREBSOZJMDO3A4KVQJ4VA, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.016197995 (548h54m55.88728104s) [noise=true , address=145.79.227.9:4367 , piece_id=XTLQ3Q6J6URPFGQ33SR2L6BTQF4PCTNJBGDAPBFJ2ANEYNG7CO5Q, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.029142573 (548h54m55.900225619s) [noise=true , address=145.79.227.36:4519 , piece_id=MXI7X5SKSVRVYRAJWQQSR2JXEGKOT664G5UV36BJ36PORUNVLVSQ, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.029656482 (548h54m55.900739517s) [noise=true , address=145.79.227.16:4911 , piece_id=U5MAVG64WWGY7VXKJHJ6XVWO7DTKR37VMCLHDL4HKOSPSPGMYLKA, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.030455133 (548h54m55.901538168s) [noise=true , address=145.79.227.50:4374 , piece_id=Q6VGBXXCDR53FPI25VJMXGG4YKBJPYXMMAACNDP6VOTEKQZ43QKA, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.031054133 (548h54m55.902137169s) [noise=true , address=145.79.227.20:4439 , piece_id=OEQOKZUV6PF5PVCMB6K7O7E2ADGKLF7YYJXNETJAOL7JIL7LQYSA, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.034831236 (548h54m55.905914281s) [noise=true , address=145.79.227.39:4015 , piece_id=ZGICJN6ZHBJVKIGF5JRAPK7K4LIMGVRRBQUAC5TL23DA2AVUZMAQ, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.039208280 (548h54m55.910291325s) [noise=true , address=145.79.227.2:4114 , piece_id=2F4ST2WYZVQQ4CDE57K47D3XMPJNO3YJNBLPPJAOQFYYRQAVWUDQ, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.050011281 (548h54m55.921094326s) [noise=true , address=145.79.227.7:4294 , piece_id=K54RV6XXNZUTCG343DZLRVBH3SB45URWFU6ZTHRWR3ZQGQV2RM7Q, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.175578192 (548h54m56.046661227s) [noise=true , address=77.90.31.11:4553 , piece_id=4GGISQWLDH7M7YZZIJHY3X5PXYFZDCNJSS4JGYMEDQDOXNLURP6A, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.187145397 (548h54m56.058228442s) [noise=true , address=77.90.31.19:4844 , piece_id=UY6CQUAWGN2TQWNB3PB6MRLEQ7WWD7O3C3DEIT5LCYTRSE6BZVJQ, seg_pos={0 0}, upload=481306]: Done uploading piece. err: <nil> Dec 28 12:23:56.187176126 (548h54m56.058259171s) [seg_pos={0 0}, upload=481306]: Segment reached optimal threshold of 13 pieces. Dec 28 12:23:56.187425280 (548h54m56.058508325s) [noise=true , address=45.91.116.12:4228 , piece_id=6K2V43NFIGDFCPIYC2NRYNCWOYGPABDE5IJ427HQLGS3ZR43M2JA, seg_pos={0 0}, upload=481306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 12:23:56.187434518 (548h54m56.058517554s) [noise=true , address=45.91.116.12:4228 , piece_id=6K2V43NFIGDFCPIYC2NRYNCWOYGPABDE5IJ427HQLGS3ZR43M2JA, seg_pos={0 0}, upload=481306]: Long tail cancellation due to optimal threshold Dec 28 12:23:56.187666469 (548h54m56.058749525s) [noise=true , address=77.90.31.10:4517 , piece_id=FVJMTGTEL6OCPGX3EQ62FCTXDZXNEZTXNOCA3OHREPO3SODUM6DQ, seg_pos={0 0}, upload=481306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 12:23:56.187678583 (548h54m56.058761638s) [noise=true , address=77.90.31.10:4517 , piece_id=FVJMTGTEL6OCPGX3EQ62FCTXDZXNEZTXNOCA3OHREPO3SODUM6DQ, seg_pos={0 0}, upload=481306]: Long tail cancellation due to optimal threshold Dec 28 12:23:56.187699883 (548h54m56.058782918s) [noise=true , address=45.91.116.9:4263 , piece_id=WEYPS25VOPTMTSEJL6VMLWIV6R36MVOKYEFZMGQJCXCRABUIZGQQ, seg_pos={0 0}, upload=481306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 12:23:56.187708019 (548h54m56.058791054s) [noise=true , address=45.91.116.9:4263 , piece_id=WEYPS25VOPTMTSEJL6VMLWIV6R36MVOKYEFZMGQJCXCRABUIZGQQ, seg_pos={0 0}, upload=481306]: Long tail cancellation due to optimal threshold Dec 28 12:23:56.187727917 (548h54m56.058810962s) [seg_pos={0 0}, upload=481306]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:23:56.187814131 (548h54m56.058897186s) [upload=481306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:23:56.187821545 (548h54m56.05890458s) [upload=481306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:23:56.466488795 (548h54m56.33757183s) [upload=481306]: Done uploading segments... Dec 28 12:23:56.466517449 (548h54m56.337600494s) [upload=481306]: Upload finished. err: <nil> Duration: 30.002841984s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 18.470012594s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:24:15 UTC) orbiter warning: context deadline exceeded Duration: 18.470038184s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X] duration: 5.000481026s trace: https://jaeger.dev.storj.io/trace/3b5ae53663861f01 FAIL (2025-12-28 12:23:41 UTC) orbiter warning: context deadline exceeded Duration: 5.000519649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XX...XXX.......X..XX....X...X.........X..XX.X....XX.X.....] duration: 6.333371818s trace: https://jaeger.dev.storj.io/trace/7c57815d0a171cf9 FAIL (2025-12-28 12:23:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 481233]: Starting upload Dec 28 12:18:52.238771198 (548h49m52.109854233s) [upload=481233]: Done starting upload Dec 28 12:18:52.238958314 (548h49m52.110041359s) [upload=481233]: Uploading segments... Dec 28 12:18:52.239003010 (548h49m52.110086045s) [upload=481233]: Got next segment. Inline: false Dec 28 12:18:52.239035562 (548h49m52.110118597s) [upload=481233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:18:52.239040752 (548h49m52.110123797s) [upload=481233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:18:52.483097267 (548h49m52.354180322s) [seg_pos={0 0}, upload=481233]: Begin upload segment... Dec 28 12:18:52.483168983 (548h49m52.354252028s) [seg_pos={0 0}, upload=481233]: Done begin upload segment. Dec 28 12:18:52.483254255 (548h49m52.3543373s) [noise=true , address=145.79.227.11:4730 , piece_id=VJDU624MCLNREPNTI5EE6OSDDXSHK3FNCIG5EQ3BMM3FROH44PJQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.483275776 (548h49m52.354358821s) [noise=true , address=77.90.31.12:4589 , piece_id=K4SAZOTYCURANXH555OBXPDYX7V3H54KXIRDRR3NIEWZN2T5UJKQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.483850180 (548h49m52.354933225s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484024932 (548h49m52.355107967s) [noise=true , address=145.79.227.36:4518 , piece_id=TIYLVCNXT5V22G4JNO5SNGXN2WGIUQC42T4NBPIUMGOOZHOVMXEA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484272063 (548h49m52.355355108s) [noise=true , address=145.79.227.14:4837 , piece_id=YNYWF2DYGZJUDGAUOM3BEHWTVSBNLDVMTKUEK3ROSPXOYZ5V3JOA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484371031 (548h49m52.355454077s) [noise=true , address=145.79.227.39:4015 , piece_id=X4U7U6I4M3RNYUQ3JUEYLWXJJKI3YAJKWU3254RD4ZWBOIBR7XYQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484709867 (548h49m52.355792912s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485134145 (548h49m52.35621718s) [noise=true , address=145.79.227.47:4266 , piece_id=LX3G3W3TW5AHFMC2S6DEREUVCBYWTUBTPYVXNVAFADHBRMLTDTDQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485279792 (548h49m52.356362837s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485594752 (548h49m52.356677797s) [noise=true , address=77.90.31.16:4735 , piece_id=E3ZKEFAX2RLO7KDHIP7AAHSHOGQN2QUAYQX45OC7ZEAV34E7SZ2Q, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485936052 (548h49m52.357019087s) [noise=true , address=145.79.227.37:4567 , piece_id=D5AA7D4CZ52YRFWVJWNOZX7H56MN4T6OYIGE4XJ7YM5TMHQEL3TA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486280808 (548h49m52.357363843s) [noise=true , address=145.79.227.41:4086 , piece_id=YZV4RCNE4PG2CVCON6XJ4AKOI2HDA6R7E2KZ763GS6W4PS25F3EA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486578906 (548h49m52.357661941s) [noise=true , address=45.91.116.14:4190 , piece_id=7S5DAQFQ626DWSZJS4R3JPDPFP7SABVBRNY7CBNPA4XGRUFVRHZA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486764008 (548h49m52.357847053s) [noise=true , address=145.79.227.16:4910 , piece_id=A6RIE25A3EETEDHNLFSAQVYZYKXNMQAPBISZMYYRA3CAKPQFPEKA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.487255674 (548h49m52.35833871s) [noise=true , address=45.91.116.13:4155 , piece_id=K4KJZXB225ZI7SPFD3C6VIWDOU7LKXEZGG5NO77IENMNEXO3KJ7A, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.487350886 (548h49m52.358433921s) [noise=true , address=145.79.227.15:4874 , piece_id=AMS2T2JI73PGUPCGGJUQVASGW56WP54RZ7DR2IELHQJ5Z65AXVOA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:54.203727189 (548h49m54.074810234s) [upload=481233]: Next returned nil segment Dec 28 12:18:54.203741315 (548h49m54.07482436s) [upload=481233]: Waiting for error group managing segments... Dec 28 12:18:54.477655054 (548h49m54.348738099s) [noise=true , address=145.79.227.11:4730 , piece_id=VJDU624MCLNREPNTI5EE6OSDDXSHK3FNCIG5EQ3BMM3FROH44PJQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.491300388 (548h49m54.362383423s) [noise=true , address=145.79.227.15:4874 , piece_id=AMS2T2JI73PGUPCGGJUQVASGW56WP54RZ7DR2IELHQJ5Z65AXVOA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.493726196 (548h49m54.364809231s) [noise=true , address=145.79.227.16:4910 , piece_id=A6RIE25A3EETEDHNLFSAQVYZYKXNMQAPBISZMYYRA3CAKPQFPEKA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.494469683 (548h49m54.365552718s) [noise=true , address=145.79.227.36:4518 , piece_id=TIYLVCNXT5V22G4JNO5SNGXN2WGIUQC42T4NBPIUMGOOZHOVMXEA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.495038846 (548h49m54.366121891s) [noise=true , address=145.79.227.39:4015 , piece_id=X4U7U6I4M3RNYUQ3JUEYLWXJJKI3YAJKWU3254RD4ZWBOIBR7XYQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.500442425 (548h49m54.37152547s) [noise=true , address=145.79.227.47:4266 , piece_id=LX3G3W3TW5AHFMC2S6DEREUVCBYWTUBTPYVXNVAFADHBRMLTDTDQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.503625106 (548h49m54.374708141s) [noise=true , address=145.79.227.37:4567 , piece_id=D5AA7D4CZ52YRFWVJWNOZX7H56MN4T6OYIGE4XJ7YM5TMHQEL3TA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.511969214 (548h49m54.383052259s) [noise=true , address=145.79.227.14:4837 , piece_id=YNYWF2DYGZJUDGAUOM3BEHWTVSBNLDVMTKUEK3ROSPXOYZ5V3JOA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.513272565 (548h49m54.384355601s) [noise=true , address=145.79.227.41:4086 , piece_id=YZV4RCNE4PG2CVCON6XJ4AKOI2HDA6R7E2KZ763GS6W4PS25F3EA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.732481216 (548h49m54.603564252s) [noise=true , address=45.91.116.14:4190 , piece_id=7S5DAQFQ626DWSZJS4R3JPDPFP7SABVBRNY7CBNPA4XGRUFVRHZA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.792501676 (548h49m54.663584721s) [noise=true , address=77.90.31.12:4589 , piece_id=K4SAZOTYCURANXH555OBXPDYX7V3H54KXIRDRR3NIEWZN2T5UJKQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.821467191 (548h49m54.692550226s) [noise=true , address=77.90.31.16:4735 , piece_id=E3ZKEFAX2RLO7KDHIP7AAHSHOGQN2QUAYQX45OC7ZEAV34E7SZ2Q, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.843755331 (548h49m54.714838376s) [noise=true , address=45.91.116.13:4155 , piece_id=K4KJZXB225ZI7SPFD3C6VIWDOU7LKXEZGG5NO77IENMNEXO3KJ7A, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.843780700 (548h49m54.714863745s) [seg_pos={0 0}, upload=481233]: Segment reached optimal threshold of 13 pieces. Dec 28 12:18:54.843968487 (548h49m54.715051532s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 12:18:54.843975049 (548h49m54.715058084s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844138220 (548h49m54.715221255s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection Dec 28 12:18:54.844144782 (548h49m54.715227828s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844607825 (548h49m54.71569087s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: read tcp 188.166.225.170:44982->77.90.31.19:4842: use of closed network connection; conn closed: op Dec 28 12:18:54.844625598 (548h49m54.715708643s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844653872 (548h49m54.715736917s) [seg_pos={0 0}, upload=481233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:18:54.844793698 (548h49m54.715876744s) [upload=481233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:18:54.844799469 (548h49m54.715882504s) [upload=481233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:18:55.136407892 (548h49m55.007490927s) [upload=481233]: Done uploading segments... Dec 28 12:18:55.136430835 (548h49m55.00751388s) [upload=481233]: Upload finished. err: <nil> Duration: 6.333396924s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 20.696813794s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:23:12 UTC) orbiter warning: context deadline exceeded Duration: 20.696839683s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.] duration: 3.517572016s trace: https://jaeger.dev.storj.io/trace/61fa771c57a4edf4 FAIL (2025-12-28 12:23:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517598846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...XX...XXX.......X..XX....X...X.........X..XX.X....XX.X....] duration: 5.723787616s trace: https://jaeger.dev.storj.io/trace/7c57815d0a171cf9 FAIL (2025-12-28 12:22:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 481233]: Starting upload Dec 28 12:18:52.238771198 (548h49m52.109854233s) [upload=481233]: Done starting upload Dec 28 12:18:52.238958314 (548h49m52.110041359s) [upload=481233]: Uploading segments... Dec 28 12:18:52.239003010 (548h49m52.110086045s) [upload=481233]: Got next segment. Inline: false Dec 28 12:18:52.239035562 (548h49m52.110118597s) [upload=481233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:18:52.239040752 (548h49m52.110123797s) [upload=481233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:18:52.483097267 (548h49m52.354180322s) [seg_pos={0 0}, upload=481233]: Begin upload segment... Dec 28 12:18:52.483168983 (548h49m52.354252028s) [seg_pos={0 0}, upload=481233]: Done begin upload segment. Dec 28 12:18:52.483254255 (548h49m52.3543373s) [noise=true , address=145.79.227.11:4730 , piece_id=VJDU624MCLNREPNTI5EE6OSDDXSHK3FNCIG5EQ3BMM3FROH44PJQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.483275776 (548h49m52.354358821s) [noise=true , address=77.90.31.12:4589 , piece_id=K4SAZOTYCURANXH555OBXPDYX7V3H54KXIRDRR3NIEWZN2T5UJKQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.483850180 (548h49m52.354933225s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484024932 (548h49m52.355107967s) [noise=true , address=145.79.227.36:4518 , piece_id=TIYLVCNXT5V22G4JNO5SNGXN2WGIUQC42T4NBPIUMGOOZHOVMXEA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484272063 (548h49m52.355355108s) [noise=true , address=145.79.227.14:4837 , piece_id=YNYWF2DYGZJUDGAUOM3BEHWTVSBNLDVMTKUEK3ROSPXOYZ5V3JOA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484371031 (548h49m52.355454077s) [noise=true , address=145.79.227.39:4015 , piece_id=X4U7U6I4M3RNYUQ3JUEYLWXJJKI3YAJKWU3254RD4ZWBOIBR7XYQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484709867 (548h49m52.355792912s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485134145 (548h49m52.35621718s) [noise=true , address=145.79.227.47:4266 , piece_id=LX3G3W3TW5AHFMC2S6DEREUVCBYWTUBTPYVXNVAFADHBRMLTDTDQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485279792 (548h49m52.356362837s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485594752 (548h49m52.356677797s) [noise=true , address=77.90.31.16:4735 , piece_id=E3ZKEFAX2RLO7KDHIP7AAHSHOGQN2QUAYQX45OC7ZEAV34E7SZ2Q, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485936052 (548h49m52.357019087s) [noise=true , address=145.79.227.37:4567 , piece_id=D5AA7D4CZ52YRFWVJWNOZX7H56MN4T6OYIGE4XJ7YM5TMHQEL3TA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486280808 (548h49m52.357363843s) [noise=true , address=145.79.227.41:4086 , piece_id=YZV4RCNE4PG2CVCON6XJ4AKOI2HDA6R7E2KZ763GS6W4PS25F3EA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486578906 (548h49m52.357661941s) [noise=true , address=45.91.116.14:4190 , piece_id=7S5DAQFQ626DWSZJS4R3JPDPFP7SABVBRNY7CBNPA4XGRUFVRHZA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486764008 (548h49m52.357847053s) [noise=true , address=145.79.227.16:4910 , piece_id=A6RIE25A3EETEDHNLFSAQVYZYKXNMQAPBISZMYYRA3CAKPQFPEKA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.487255674 (548h49m52.35833871s) [noise=true , address=45.91.116.13:4155 , piece_id=K4KJZXB225ZI7SPFD3C6VIWDOU7LKXEZGG5NO77IENMNEXO3KJ7A, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.487350886 (548h49m52.358433921s) [noise=true , address=145.79.227.15:4874 , piece_id=AMS2T2JI73PGUPCGGJUQVASGW56WP54RZ7DR2IELHQJ5Z65AXVOA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:54.203727189 (548h49m54.074810234s) [upload=481233]: Next returned nil segment Dec 28 12:18:54.203741315 (548h49m54.07482436s) [upload=481233]: Waiting for error group managing segments... Dec 28 12:18:54.477655054 (548h49m54.348738099s) [noise=true , address=145.79.227.11:4730 , piece_id=VJDU624MCLNREPNTI5EE6OSDDXSHK3FNCIG5EQ3BMM3FROH44PJQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.491300388 (548h49m54.362383423s) [noise=true , address=145.79.227.15:4874 , piece_id=AMS2T2JI73PGUPCGGJUQVASGW56WP54RZ7DR2IELHQJ5Z65AXVOA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.493726196 (548h49m54.364809231s) [noise=true , address=145.79.227.16:4910 , piece_id=A6RIE25A3EETEDHNLFSAQVYZYKXNMQAPBISZMYYRA3CAKPQFPEKA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.494469683 (548h49m54.365552718s) [noise=true , address=145.79.227.36:4518 , piece_id=TIYLVCNXT5V22G4JNO5SNGXN2WGIUQC42T4NBPIUMGOOZHOVMXEA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.495038846 (548h49m54.366121891s) [noise=true , address=145.79.227.39:4015 , piece_id=X4U7U6I4M3RNYUQ3JUEYLWXJJKI3YAJKWU3254RD4ZWBOIBR7XYQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.500442425 (548h49m54.37152547s) [noise=true , address=145.79.227.47:4266 , piece_id=LX3G3W3TW5AHFMC2S6DEREUVCBYWTUBTPYVXNVAFADHBRMLTDTDQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.503625106 (548h49m54.374708141s) [noise=true , address=145.79.227.37:4567 , piece_id=D5AA7D4CZ52YRFWVJWNOZX7H56MN4T6OYIGE4XJ7YM5TMHQEL3TA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.511969214 (548h49m54.383052259s) [noise=true , address=145.79.227.14:4837 , piece_id=YNYWF2DYGZJUDGAUOM3BEHWTVSBNLDVMTKUEK3ROSPXOYZ5V3JOA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.513272565 (548h49m54.384355601s) [noise=true , address=145.79.227.41:4086 , piece_id=YZV4RCNE4PG2CVCON6XJ4AKOI2HDA6R7E2KZ763GS6W4PS25F3EA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.732481216 (548h49m54.603564252s) [noise=true , address=45.91.116.14:4190 , piece_id=7S5DAQFQ626DWSZJS4R3JPDPFP7SABVBRNY7CBNPA4XGRUFVRHZA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.792501676 (548h49m54.663584721s) [noise=true , address=77.90.31.12:4589 , piece_id=K4SAZOTYCURANXH555OBXPDYX7V3H54KXIRDRR3NIEWZN2T5UJKQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.821467191 (548h49m54.692550226s) [noise=true , address=77.90.31.16:4735 , piece_id=E3ZKEFAX2RLO7KDHIP7AAHSHOGQN2QUAYQX45OC7ZEAV34E7SZ2Q, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.843755331 (548h49m54.714838376s) [noise=true , address=45.91.116.13:4155 , piece_id=K4KJZXB225ZI7SPFD3C6VIWDOU7LKXEZGG5NO77IENMNEXO3KJ7A, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.843780700 (548h49m54.714863745s) [seg_pos={0 0}, upload=481233]: Segment reached optimal threshold of 13 pieces. Dec 28 12:18:54.843968487 (548h49m54.715051532s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 12:18:54.843975049 (548h49m54.715058084s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844138220 (548h49m54.715221255s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection Dec 28 12:18:54.844144782 (548h49m54.715227828s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844607825 (548h49m54.71569087s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: read tcp 188.166.225.170:44982->77.90.31.19:4842: use of closed network connection; conn closed: op Dec 28 12:18:54.844625598 (548h49m54.715708643s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844653872 (548h49m54.715736917s) [seg_pos={0 0}, upload=481233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:18:54.844793698 (548h49m54.715876744s) [upload=481233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:18:54.844799469 (548h49m54.715882504s) [upload=481233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:18:55.136407892 (548h49m55.007490927s) [upload=481233]: Done uploading segments... Dec 28 12:18:55.136430835 (548h49m55.00751388s) [upload=481233]: Upload finished. err: <nil> Duration: 5.723817102s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 20.634104454s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:22:34 UTC) orbiter warning: context deadline exceeded Duration: 20.634130774s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 16.772133746s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:22:06 UTC) orbiter warning: context deadline exceeded Duration: 16.772164934s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.XX.XX] duration: 5.001156694s trace: https://jaeger.dev.storj.io/trace/61fa771c57a4edf4 FAIL (2025-12-28 12:21:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00118131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....XX...XXX.......X..XX....X...X.........X..XX.X....XX.X...] duration: 5.943418549s trace: https://jaeger.dev.storj.io/trace/7c57815d0a171cf9 FAIL (2025-12-28 12:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 481233]: Starting upload Dec 28 12:18:52.238771198 (548h49m52.109854233s) [upload=481233]: Done starting upload Dec 28 12:18:52.238958314 (548h49m52.110041359s) [upload=481233]: Uploading segments... Dec 28 12:18:52.239003010 (548h49m52.110086045s) [upload=481233]: Got next segment. Inline: false Dec 28 12:18:52.239035562 (548h49m52.110118597s) [upload=481233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:18:52.239040752 (548h49m52.110123797s) [upload=481233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:18:52.483097267 (548h49m52.354180322s) [seg_pos={0 0}, upload=481233]: Begin upload segment... Dec 28 12:18:52.483168983 (548h49m52.354252028s) [seg_pos={0 0}, upload=481233]: Done begin upload segment. Dec 28 12:18:52.483254255 (548h49m52.3543373s) [noise=true , address=145.79.227.11:4730 , piece_id=VJDU624MCLNREPNTI5EE6OSDDXSHK3FNCIG5EQ3BMM3FROH44PJQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.483275776 (548h49m52.354358821s) [noise=true , address=77.90.31.12:4589 , piece_id=K4SAZOTYCURANXH555OBXPDYX7V3H54KXIRDRR3NIEWZN2T5UJKQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.483850180 (548h49m52.354933225s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484024932 (548h49m52.355107967s) [noise=true , address=145.79.227.36:4518 , piece_id=TIYLVCNXT5V22G4JNO5SNGXN2WGIUQC42T4NBPIUMGOOZHOVMXEA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484272063 (548h49m52.355355108s) [noise=true , address=145.79.227.14:4837 , piece_id=YNYWF2DYGZJUDGAUOM3BEHWTVSBNLDVMTKUEK3ROSPXOYZ5V3JOA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484371031 (548h49m52.355454077s) [noise=true , address=145.79.227.39:4015 , piece_id=X4U7U6I4M3RNYUQ3JUEYLWXJJKI3YAJKWU3254RD4ZWBOIBR7XYQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484709867 (548h49m52.355792912s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485134145 (548h49m52.35621718s) [noise=true , address=145.79.227.47:4266 , piece_id=LX3G3W3TW5AHFMC2S6DEREUVCBYWTUBTPYVXNVAFADHBRMLTDTDQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485279792 (548h49m52.356362837s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485594752 (548h49m52.356677797s) [noise=true , address=77.90.31.16:4735 , piece_id=E3ZKEFAX2RLO7KDHIP7AAHSHOGQN2QUAYQX45OC7ZEAV34E7SZ2Q, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485936052 (548h49m52.357019087s) [noise=true , address=145.79.227.37:4567 , piece_id=D5AA7D4CZ52YRFWVJWNOZX7H56MN4T6OYIGE4XJ7YM5TMHQEL3TA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486280808 (548h49m52.357363843s) [noise=true , address=145.79.227.41:4086 , piece_id=YZV4RCNE4PG2CVCON6XJ4AKOI2HDA6R7E2KZ763GS6W4PS25F3EA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486578906 (548h49m52.357661941s) [noise=true , address=45.91.116.14:4190 , piece_id=7S5DAQFQ626DWSZJS4R3JPDPFP7SABVBRNY7CBNPA4XGRUFVRHZA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486764008 (548h49m52.357847053s) [noise=true , address=145.79.227.16:4910 , piece_id=A6RIE25A3EETEDHNLFSAQVYZYKXNMQAPBISZMYYRA3CAKPQFPEKA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.487255674 (548h49m52.35833871s) [noise=true , address=45.91.116.13:4155 , piece_id=K4KJZXB225ZI7SPFD3C6VIWDOU7LKXEZGG5NO77IENMNEXO3KJ7A, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.487350886 (548h49m52.358433921s) [noise=true , address=145.79.227.15:4874 , piece_id=AMS2T2JI73PGUPCGGJUQVASGW56WP54RZ7DR2IELHQJ5Z65AXVOA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:54.203727189 (548h49m54.074810234s) [upload=481233]: Next returned nil segment Dec 28 12:18:54.203741315 (548h49m54.07482436s) [upload=481233]: Waiting for error group managing segments... Dec 28 12:18:54.477655054 (548h49m54.348738099s) [noise=true , address=145.79.227.11:4730 , piece_id=VJDU624MCLNREPNTI5EE6OSDDXSHK3FNCIG5EQ3BMM3FROH44PJQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.491300388 (548h49m54.362383423s) [noise=true , address=145.79.227.15:4874 , piece_id=AMS2T2JI73PGUPCGGJUQVASGW56WP54RZ7DR2IELHQJ5Z65AXVOA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.493726196 (548h49m54.364809231s) [noise=true , address=145.79.227.16:4910 , piece_id=A6RIE25A3EETEDHNLFSAQVYZYKXNMQAPBISZMYYRA3CAKPQFPEKA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.494469683 (548h49m54.365552718s) [noise=true , address=145.79.227.36:4518 , piece_id=TIYLVCNXT5V22G4JNO5SNGXN2WGIUQC42T4NBPIUMGOOZHOVMXEA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.495038846 (548h49m54.366121891s) [noise=true , address=145.79.227.39:4015 , piece_id=X4U7U6I4M3RNYUQ3JUEYLWXJJKI3YAJKWU3254RD4ZWBOIBR7XYQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.500442425 (548h49m54.37152547s) [noise=true , address=145.79.227.47:4266 , piece_id=LX3G3W3TW5AHFMC2S6DEREUVCBYWTUBTPYVXNVAFADHBRMLTDTDQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.503625106 (548h49m54.374708141s) [noise=true , address=145.79.227.37:4567 , piece_id=D5AA7D4CZ52YRFWVJWNOZX7H56MN4T6OYIGE4XJ7YM5TMHQEL3TA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.511969214 (548h49m54.383052259s) [noise=true , address=145.79.227.14:4837 , piece_id=YNYWF2DYGZJUDGAUOM3BEHWTVSBNLDVMTKUEK3ROSPXOYZ5V3JOA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.513272565 (548h49m54.384355601s) [noise=true , address=145.79.227.41:4086 , piece_id=YZV4RCNE4PG2CVCON6XJ4AKOI2HDA6R7E2KZ763GS6W4PS25F3EA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.732481216 (548h49m54.603564252s) [noise=true , address=45.91.116.14:4190 , piece_id=7S5DAQFQ626DWSZJS4R3JPDPFP7SABVBRNY7CBNPA4XGRUFVRHZA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.792501676 (548h49m54.663584721s) [noise=true , address=77.90.31.12:4589 , piece_id=K4SAZOTYCURANXH555OBXPDYX7V3H54KXIRDRR3NIEWZN2T5UJKQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.821467191 (548h49m54.692550226s) [noise=true , address=77.90.31.16:4735 , piece_id=E3ZKEFAX2RLO7KDHIP7AAHSHOGQN2QUAYQX45OC7ZEAV34E7SZ2Q, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.843755331 (548h49m54.714838376s) [noise=true , address=45.91.116.13:4155 , piece_id=K4KJZXB225ZI7SPFD3C6VIWDOU7LKXEZGG5NO77IENMNEXO3KJ7A, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.843780700 (548h49m54.714863745s) [seg_pos={0 0}, upload=481233]: Segment reached optimal threshold of 13 pieces. Dec 28 12:18:54.843968487 (548h49m54.715051532s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 12:18:54.843975049 (548h49m54.715058084s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844138220 (548h49m54.715221255s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection Dec 28 12:18:54.844144782 (548h49m54.715227828s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844607825 (548h49m54.71569087s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: read tcp 188.166.225.170:44982->77.90.31.19:4842: use of closed network connection; conn closed: op Dec 28 12:18:54.844625598 (548h49m54.715708643s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844653872 (548h49m54.715736917s) [seg_pos={0 0}, upload=481233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:18:54.844793698 (548h49m54.715876744s) [upload=481233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:18:54.844799469 (548h49m54.715882504s) [upload=481233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:18:55.136407892 (548h49m55.007490927s) [upload=481233]: Done uploading segments... Dec 28 12:18:55.136430835 (548h49m55.00751388s) [upload=481233]: Upload finished. err: <nil> Duration: 5.943445781s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration: 11.115127161s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:21:18 UTC) orbiter warning: context deadline exceeded Duration: 11.115159864s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....XX...XXX.......X..XX....X...X.........X..XX.X....XX.X..] duration: 5.951940676s trace: https://jaeger.dev.storj.io/trace/7c57815d0a171cf9 FAIL (2025-12-28 12:21:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 481233]: Starting upload Dec 28 12:18:52.238771198 (548h49m52.109854233s) [upload=481233]: Done starting upload Dec 28 12:18:52.238958314 (548h49m52.110041359s) [upload=481233]: Uploading segments... Dec 28 12:18:52.239003010 (548h49m52.110086045s) [upload=481233]: Got next segment. Inline: false Dec 28 12:18:52.239035562 (548h49m52.110118597s) [upload=481233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:18:52.239040752 (548h49m52.110123797s) [upload=481233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:18:52.483097267 (548h49m52.354180322s) [seg_pos={0 0}, upload=481233]: Begin upload segment... Dec 28 12:18:52.483168983 (548h49m52.354252028s) [seg_pos={0 0}, upload=481233]: Done begin upload segment. Dec 28 12:18:52.483254255 (548h49m52.3543373s) [noise=true , address=145.79.227.11:4730 , piece_id=VJDU624MCLNREPNTI5EE6OSDDXSHK3FNCIG5EQ3BMM3FROH44PJQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.483275776 (548h49m52.354358821s) [noise=true , address=77.90.31.12:4589 , piece_id=K4SAZOTYCURANXH555OBXPDYX7V3H54KXIRDRR3NIEWZN2T5UJKQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.483850180 (548h49m52.354933225s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484024932 (548h49m52.355107967s) [noise=true , address=145.79.227.36:4518 , piece_id=TIYLVCNXT5V22G4JNO5SNGXN2WGIUQC42T4NBPIUMGOOZHOVMXEA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484272063 (548h49m52.355355108s) [noise=true , address=145.79.227.14:4837 , piece_id=YNYWF2DYGZJUDGAUOM3BEHWTVSBNLDVMTKUEK3ROSPXOYZ5V3JOA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484371031 (548h49m52.355454077s) [noise=true , address=145.79.227.39:4015 , piece_id=X4U7U6I4M3RNYUQ3JUEYLWXJJKI3YAJKWU3254RD4ZWBOIBR7XYQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.484709867 (548h49m52.355792912s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485134145 (548h49m52.35621718s) [noise=true , address=145.79.227.47:4266 , piece_id=LX3G3W3TW5AHFMC2S6DEREUVCBYWTUBTPYVXNVAFADHBRMLTDTDQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485279792 (548h49m52.356362837s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485594752 (548h49m52.356677797s) [noise=true , address=77.90.31.16:4735 , piece_id=E3ZKEFAX2RLO7KDHIP7AAHSHOGQN2QUAYQX45OC7ZEAV34E7SZ2Q, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.485936052 (548h49m52.357019087s) [noise=true , address=145.79.227.37:4567 , piece_id=D5AA7D4CZ52YRFWVJWNOZX7H56MN4T6OYIGE4XJ7YM5TMHQEL3TA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486280808 (548h49m52.357363843s) [noise=true , address=145.79.227.41:4086 , piece_id=YZV4RCNE4PG2CVCON6XJ4AKOI2HDA6R7E2KZ763GS6W4PS25F3EA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486578906 (548h49m52.357661941s) [noise=true , address=45.91.116.14:4190 , piece_id=7S5DAQFQ626DWSZJS4R3JPDPFP7SABVBRNY7CBNPA4XGRUFVRHZA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.486764008 (548h49m52.357847053s) [noise=true , address=145.79.227.16:4910 , piece_id=A6RIE25A3EETEDHNLFSAQVYZYKXNMQAPBISZMYYRA3CAKPQFPEKA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.487255674 (548h49m52.35833871s) [noise=true , address=45.91.116.13:4155 , piece_id=K4KJZXB225ZI7SPFD3C6VIWDOU7LKXEZGG5NO77IENMNEXO3KJ7A, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:52.487350886 (548h49m52.358433921s) [noise=true , address=145.79.227.15:4874 , piece_id=AMS2T2JI73PGUPCGGJUQVASGW56WP54RZ7DR2IELHQJ5Z65AXVOA, seg_pos={0 0}, upload=481233]: Uploading piece... Dec 28 12:18:54.203727189 (548h49m54.074810234s) [upload=481233]: Next returned nil segment Dec 28 12:18:54.203741315 (548h49m54.07482436s) [upload=481233]: Waiting for error group managing segments... Dec 28 12:18:54.477655054 (548h49m54.348738099s) [noise=true , address=145.79.227.11:4730 , piece_id=VJDU624MCLNREPNTI5EE6OSDDXSHK3FNCIG5EQ3BMM3FROH44PJQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.491300388 (548h49m54.362383423s) [noise=true , address=145.79.227.15:4874 , piece_id=AMS2T2JI73PGUPCGGJUQVASGW56WP54RZ7DR2IELHQJ5Z65AXVOA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.493726196 (548h49m54.364809231s) [noise=true , address=145.79.227.16:4910 , piece_id=A6RIE25A3EETEDHNLFSAQVYZYKXNMQAPBISZMYYRA3CAKPQFPEKA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.494469683 (548h49m54.365552718s) [noise=true , address=145.79.227.36:4518 , piece_id=TIYLVCNXT5V22G4JNO5SNGXN2WGIUQC42T4NBPIUMGOOZHOVMXEA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.495038846 (548h49m54.366121891s) [noise=true , address=145.79.227.39:4015 , piece_id=X4U7U6I4M3RNYUQ3JUEYLWXJJKI3YAJKWU3254RD4ZWBOIBR7XYQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.500442425 (548h49m54.37152547s) [noise=true , address=145.79.227.47:4266 , piece_id=LX3G3W3TW5AHFMC2S6DEREUVCBYWTUBTPYVXNVAFADHBRMLTDTDQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.503625106 (548h49m54.374708141s) [noise=true , address=145.79.227.37:4567 , piece_id=D5AA7D4CZ52YRFWVJWNOZX7H56MN4T6OYIGE4XJ7YM5TMHQEL3TA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.511969214 (548h49m54.383052259s) [noise=true , address=145.79.227.14:4837 , piece_id=YNYWF2DYGZJUDGAUOM3BEHWTVSBNLDVMTKUEK3ROSPXOYZ5V3JOA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.513272565 (548h49m54.384355601s) [noise=true , address=145.79.227.41:4086 , piece_id=YZV4RCNE4PG2CVCON6XJ4AKOI2HDA6R7E2KZ763GS6W4PS25F3EA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.732481216 (548h49m54.603564252s) [noise=true , address=45.91.116.14:4190 , piece_id=7S5DAQFQ626DWSZJS4R3JPDPFP7SABVBRNY7CBNPA4XGRUFVRHZA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.792501676 (548h49m54.663584721s) [noise=true , address=77.90.31.12:4589 , piece_id=K4SAZOTYCURANXH555OBXPDYX7V3H54KXIRDRR3NIEWZN2T5UJKQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.821467191 (548h49m54.692550226s) [noise=true , address=77.90.31.16:4735 , piece_id=E3ZKEFAX2RLO7KDHIP7AAHSHOGQN2QUAYQX45OC7ZEAV34E7SZ2Q, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.843755331 (548h49m54.714838376s) [noise=true , address=45.91.116.13:4155 , piece_id=K4KJZXB225ZI7SPFD3C6VIWDOU7LKXEZGG5NO77IENMNEXO3KJ7A, seg_pos={0 0}, upload=481233]: Done uploading piece. err: <nil> Dec 28 12:18:54.843780700 (548h49m54.714863745s) [seg_pos={0 0}, upload=481233]: Segment reached optimal threshold of 13 pieces. Dec 28 12:18:54.843968487 (548h49m54.715051532s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 12:18:54.843975049 (548h49m54.715058084s) [noise=true , address=45.91.116.6:4976 , piece_id=EQW2GTV6Q7VZ7LQHPNB3W7VCWMHINPXZD5USYE7LZG3TDWEGEIZA, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844138220 (548h49m54.715221255s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58360->77.90.31.10:4517: use of closed network connection Dec 28 12:18:54.844144782 (548h49m54.715227828s) [noise=true , address=77.90.31.10:4517 , piece_id=TGJ6JRK3RZFADJJTZ2BOE3OBWFYCOS3FOQPBC2GPYMZEXNNJIATQ, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844607825 (548h49m54.71569087s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: read tcp 188.166.225.170:44982->77.90.31.19:4842: use of closed network connection; conn closed: op Dec 28 12:18:54.844625598 (548h49m54.715708643s) [noise=true , address=77.90.31.19:4842 , piece_id=765VBDIV3OCSGIGRWMYDPTTIM3RNCHLIJ3XSKONWJBKTSWVACAHQ, seg_pos={0 0}, upload=481233]: Long tail cancellation due to optimal threshold Dec 28 12:18:54.844653872 (548h49m54.715736917s) [seg_pos={0 0}, upload=481233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:18:54.844793698 (548h49m54.715876744s) [upload=481233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:18:54.844799469 (548h49m54.715882504s) [upload=481233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:18:55.136407892 (548h49m55.007490927s) [upload=481233]: Done uploading segments... Dec 28 12:18:55.136430835 (548h49m55.00751388s) [upload=481233]: Upload finished. err: <nil> Duration: 5.95196849s core uplink us-select-1

1
...
152
153
154
155
156
...
6826