Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 504773 reports (Page 5558 of 10096)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................................XX....X..................] duration: 5.801913384s trace: https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2 FAIL (2025-10-31 21:42:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.802073669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X..............................XX.X.................] duration: 6.018174049s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:42:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 6.018208926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X......XX..................] duration: 5.87664069s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:41:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 5.876670908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................XX......X...................] duration: 3.48147449s trace: https://jaeger.dev.storj.io/trace/d551b29a119bb FAIL (2025-10-31 21:41:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481512774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X..............................XX.X................] duration: 6.072455314s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:41:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 6.072483488s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X......XX.................] duration: 6.028395728s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:41:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 6.028422689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X..........XX......XX....................] duration: 6.295958621s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:41:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.295987056s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X......XX................] duration: 6.013421756s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:40:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 6.013452183s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................................XX....X.................] duration: 5.760972993s trace: https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2 FAIL (2025-10-31 21:40:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.76099216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................XX......X..................] duration: 3.549569838s trace: https://jaeger.dev.storj.io/trace/d551b29a119bb FAIL (2025-10-31 21:40:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549593884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X......XX...............] duration: 5.860593373s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 5.860631346s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X..........XX......XX...................] duration: 6.003006583s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:40:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.003080102s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X..............................XX.X...............] duration: 5.884245458s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:40:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 5.884365567s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................XX......X.................] duration: 3.461287164s trace: https://jaeger.dev.storj.io/trace/d551b29a119bb FAIL (2025-10-31 21:40:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461330315s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XX....X................] duration: 5.733476873s trace: https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2 FAIL (2025-10-31 21:40:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.73350793s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................XX......X................] duration: 3.519059302s trace: https://jaeger.dev.storj.io/trace/d551b29a119bb FAIL (2025-10-31 21:40:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.519089811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X......XX..............] duration: 5.728237301s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:39:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 5.728263721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X..........XX......XX..................] duration: 6.117694912s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:39:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.117716372s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................................XX....X...............] duration: 5.846993294s trace: https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2 FAIL (2025-10-31 21:39:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.847027197s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X..........XX......XX.................] duration: 5.998866722s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:39:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.99889106s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X..............................XX.X..............] duration: 6.208626351s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:39:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 6.208661308s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................XX......X...............] duration: 3.479344173s trace: https://jaeger.dev.storj.io/trace/d551b29a119bb FAIL (2025-10-31 21:39:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479375221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X......XX.............] duration: 5.944461543s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:39:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 5.944489295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X..............................XX.X.............] duration: 5.938619206s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:39:05 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 5.938648092s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X..........XX......XX................] duration: 6.208559132s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:38:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.208603827s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XX....X..............] duration: 5.790021369s trace: https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2 FAIL (2025-10-31 21:38:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.79004819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X..............................XX.X............] duration: 5.934448825s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:38:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 5.934480645s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................X..........XX......XX...............] duration: 6.087503908s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:38:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.087533475s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X......XX............] duration: 6.009936939s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:38:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 6.009971325s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................XX......X..............] duration: 3.433036338s trace: https://jaeger.dev.storj.io/trace/d551b29a119bb FAIL (2025-10-31 21:38:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.433081424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................X..........XX......XX..............] duration: 6.323920213s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:37:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.323948687s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................................XX......X.............] duration: 3.417966393s trace: https://jaeger.dev.storj.io/trace/d551b29a119bb FAIL (2025-10-31 21:37:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417993966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................................XX....X.............] duration: 5.68648119s trace: https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2 FAIL (2025-10-31 21:37:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.686511676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X......XX...........] duration: 5.709465345s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:37:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 5.709481666s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................X..........XX......XX.............] duration: 6.057139424s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:37:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.057168558s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X..............................XX.X...........] duration: 5.861555121s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:37:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 5.861620897s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................................XX......X............] duration: 3.447690931s trace: https://jaeger.dev.storj.io/trace/d551b29a119bb FAIL (2025-10-31 21:36:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447802694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X......XX..........] duration: 5.77735039s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:36:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 5.777377361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X..............................XX.X..........] duration: 5.984611493s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:36:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 5.984642191s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................X..........XX......XX............] duration: 6.093358288s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:36:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.093386413s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................XX....X............] duration: 5.730006099s trace: https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2 FAIL (2025-10-31 21:36:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.730034082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X..............................XX.X.........] duration: 5.808168623s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:36:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 5.808205114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X..............................XX.X........] duration: 5.925332105s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:35:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 5.925369998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X......XX.........] duration: 5.791181785s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:35:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 5.79121561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................................XX......X...........] duration: 3.414006199s trace: https://jaeger.dev.storj.io/trace/d551b29a119bb FAIL (2025-10-31 21:35:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.41403827s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X..........XX......XX...........] duration: 5.596259199s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:35:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.596291761s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................X..........XX......XX..........] duration: 6.942970323s trace: https://jaeger.dev.storj.io/trace/512979013ad3487a FAIL (2025-10-31 21:35:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.94299569s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................................XX....X...........] duration: 5.925202452s trace: https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2 FAIL (2025-10-31 21:35:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.925249251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X..............................XX.X.......] duration: 5.967898369s trace: https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a FAIL (2025-10-31 21:35:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (33h48m55.924355954s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment... Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment. Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece... Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments... Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil> Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces. Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments... Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil> Duration: 5.967928296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X......XX........] duration: 5.854353108s trace: https://jaeger.dev.storj.io/trace/15db8dc6769ded56 FAIL (2025-10-31 21:35:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded ======LOGS====== :28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece... Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments... Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil> Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments... Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil> Duration: 5.854383577s core uplink us-select-1 us1-placement-20

1
...
5556
5557
5558
5559
5560
...
10096