Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................X......XX...........................] duration:
6.169976868s trace:
https://jaeger.dev.storj.io/trace/15db8dc6769ded56
FAIL (2025-10-31 21:48:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded
======LOGS======
:28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment
Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments...
Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection
Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection
Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection
Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments...
Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil>
Duration: 6.170006736s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................XX.X...........................] duration:
5.942535561s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:48:55 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 5.94256663s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........................XX....X.............................] duration:
5.930861462s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:48:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.930891499s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................XX....X............................] duration:
6.713535278s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:48:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.71356232s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X..........XX......XX............................] duration:
6.115202385s trace:
https://jaeger.dev.storj.io/trace/512979013ad3487a
FAIL (2025-10-31 21:48:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.115228224s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................XX.X..........................] duration:
5.966687363s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:48:21 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 5.966720245s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [........................XX......X...........................] duration:
3.457134902s trace:
https://jaeger.dev.storj.io/trace/d551b29a119bb
FAIL (2025-10-31 21:48:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.457165081s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................XX....X...........................] duration:
6.549691503s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:47:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.549762558s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................X......XX..........................] duration:
6.533167446s trace:
https://jaeger.dev.storj.io/trace/15db8dc6769ded56
FAIL (2025-10-31 21:47:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded
======LOGS======
:28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment
Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments...
Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection
Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection
Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection
Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments...
Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil>
Duration: 6.533193505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X..............................XX.X.........................] duration:
5.959052735s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:47:51 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 5.959079055s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................XX....X..........................] duration:
5.876273408s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:47:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.876303926s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [............X..........XX......XX...........................] duration:
6.342593032s trace:
https://jaeger.dev.storj.io/trace/512979013ad3487a
FAIL (2025-10-31 21:47:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.34261738s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X..............................XX.X........................] duration:
6.499751216s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:47:07 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 6.499775083s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.........................XX......X..........................] duration:
3.482737334s trace:
https://jaeger.dev.storj.io/trace/d551b29a119bb
FAIL (2025-10-31 21:47:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.482753775s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................X......XX.........................] duration:
6.806152892s trace:
https://jaeger.dev.storj.io/trace/15db8dc6769ded56
FAIL (2025-10-31 21:47:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded
======LOGS======
:28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment
Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments...
Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection
Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection
Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection
Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments...
Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil>
Duration: 6.806181286s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............X..........XX......XX..........................] duration:
6.460611613s trace:
https://jaeger.dev.storj.io/trace/512979013ad3487a
FAIL (2025-10-31 21:47:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.460951289s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............................XX....X.........................] duration:
5.705001877s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:46:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.705028357s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................X......XX........................] duration:
5.905627402s trace:
https://jaeger.dev.storj.io/trace/15db8dc6769ded56
FAIL (2025-10-31 21:46:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded
======LOGS======
:28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment
Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments...
Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection
Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection
Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection
Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments...
Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil>
Duration: 5.905682767s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X..............................XX.X.......................] duration:
6.028031013s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:46:03 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 6.028063955s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............X..........XX......XX.........................] duration:
6.343027882s trace:
https://jaeger.dev.storj.io/trace/512979013ad3487a
FAIL (2025-10-31 21:46:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.343076605s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..........................XX......X.........................] duration:
3.533779949s trace:
https://jaeger.dev.storj.io/trace/d551b29a119bb
FAIL (2025-10-31 21:46:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.533844312s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............................XX....X........................] duration:
5.82362304s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:45:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.823654782s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X..............................XX.X......................] duration:
6.120881882s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:45:19 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 6.120910947s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................XX....X.......................] duration:
5.874491366s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:45:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.874524278s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............X..........XX......XX........................] duration:
6.084596556s trace:
https://jaeger.dev.storj.io/trace/512979013ad3487a
FAIL (2025-10-31 21:45:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.08462507s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X..............................XX.X.....................] duration:
6.070950904s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:44:59 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 6.070967095s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [...........................XX......X........................] duration:
3.462778175s trace:
https://jaeger.dev.storj.io/trace/d551b29a119bb
FAIL (2025-10-31 21:44:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.462924794s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................X......XX.......................] duration:
6.012091299s trace:
https://jaeger.dev.storj.io/trace/15db8dc6769ded56
FAIL (2025-10-31 21:44:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded
======LOGS======
:28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment
Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments...
Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection
Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection
Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection
Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments...
Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil>
Duration: 6.012124362s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................XX....X......................] duration:
5.673506249s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:44:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.673532188s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................X......XX......................] duration:
5.910296041s trace:
https://jaeger.dev.storj.io/trace/15db8dc6769ded56
FAIL (2025-10-31 21:44:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded
======LOGS======
:28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment
Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments...
Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection
Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection
Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection
Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments...
Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil>
Duration: 5.910344935s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X..............................XX.X....................] duration:
6.148837555s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:44:09 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 6.148868203s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................................XX....X.....................] duration:
5.68484541s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:44:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.684914372s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [............................XX......X.......................] duration:
3.392911859s trace:
https://jaeger.dev.storj.io/trace/d551b29a119bb
FAIL (2025-10-31 21:44:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.392935093s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................X..........XX......XX.......................] duration:
6.107466557s trace:
https://jaeger.dev.storj.io/trace/512979013ad3487a
FAIL (2025-10-31 21:44:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.107508505s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.............................XX......X......................] duration:
3.531108481s trace:
https://jaeger.dev.storj.io/trace/d551b29a119bb
FAIL (2025-10-31 21:43:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.531131326s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................X......XX.....................] duration:
5.950639033s trace:
https://jaeger.dev.storj.io/trace/15db8dc6769ded56
FAIL (2025-10-31 21:43:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded
======LOGS======
:28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment
Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments...
Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection
Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection
Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection
Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments...
Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil>
Duration: 5.950666104s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................X..........XX......XX......................] duration:
6.186697085s trace:
https://jaeger.dev.storj.io/trace/512979013ad3487a
FAIL (2025-10-31 21:43:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.186719468s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................................XX....X....................] duration:
5.661881654s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:43:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.661906121s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X..............................XX.X...................] duration:
6.161608272s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:42:57 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 6.161626226s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................X......XX....................] duration:
5.845065637s trace:
https://jaeger.dev.storj.io/trace/15db8dc6769ded56
FAIL (2025-10-31 21:42:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded
======LOGS======
:28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment
Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments...
Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection
Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection
Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection
Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments...
Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil>
Duration: 5.845094703s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..............................XX......X.....................] duration:
3.415350618s trace:
https://jaeger.dev.storj.io/trace/d551b29a119bb
FAIL (2025-10-31 21:42:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.415376136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................................XX....X...................] duration:
5.693118845s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:42:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.693146827s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X..............................XX.X..................] duration:
6.008800516s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:42:34 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 6.00882914s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................X......XX...................] duration:
6.000928073s trace:
https://jaeger.dev.storj.io/trace/15db8dc6769ded56
FAIL (2025-10-31 21:42:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded
======LOGS======
:28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment
Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments...
Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection
Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection
Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection
Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments...
Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil>
Duration: 6.000951738s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [...............................XX......X....................] duration:
3.400440595s trace:
https://jaeger.dev.storj.io/trace/d551b29a119bb
FAIL (2025-10-31 21:42:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.400465022s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................X..........XX......XX.....................] duration:
6.197713439s trace:
https://jaeger.dev.storj.io/trace/512979013ad3487a
FAIL (2025-10-31 21:42:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.197807238s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................................XX....X..................] duration:
5.801913384s trace:
https://jaeger.dev.storj.io/trace/7a3250b50cec1fe2
FAIL (2025-10-31 21:42:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.802073669s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X..............................XX.X.................] duration:
6.018174049s trace:
https://jaeger.dev.storj.io/trace/4253cb41f8df1b2a
FAIL (2025-10-31 21:42:00 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
(33h48m55.924355954s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:29:24.228512651 (33h48m55.924361394s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:29:24.479685241 (33h48m56.175533994s) [seg_pos={0 0}, upload=29667]: Begin upload segment...
Oct 31 21:29:24.479739333 (33h48m56.175588086s) [seg_pos={0 0}, upload=29667]: Done begin upload segment.
Oct 31 21:29:24.479770803 (33h48m56.175619556s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.479881854 (33h48m56.175730607s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480549806 (33h48m56.176398579s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.480982791 (33h48m56.176831544s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.481368546 (33h48m56.177217289s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482210278 (33h48m56.178059031s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482226970 (33h48m56.178075714s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.482820410 (33h48m56.178669153s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.483833960 (33h48m56.179682703s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.485084191 (33h48m56.180932944s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.486413652 (33h48m56.182262415s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.487418536 (33h48m56.183267289s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.488596840 (33h48m56.184445593s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.489150544 (33h48m56.184999297s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.490359345 (33h48m56.186208098s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:24.500737938 (33h48m56.196586691s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Uploading piece...
Oct 31 21:29:26.105232898 (33h48m57.801081641s) [upload=29667]: Next returned nil segment
Oct 31 21:29:26.105251683 (33h48m57.801100457s) [upload=29667]: Waiting for error group managing segments...
Oct 31 21:29:26.488964936 (33h48m58.184813679s) [noise=true , address=145.79.227.13:4803 , piece_id=K7RZO5C6AXOFX267RQB3GZ2TPCCFJAESITWJDFF7YUAM4MLISBXA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.496259435 (33h48m58.192108188s) [noise=true , address=145.79.227.32:4836 , piece_id=QTD4CSLKD6LZKAYTWRO43XF566H3UKIGBP6MXBE3LYGEF3PQLL6Q, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.502088576 (33h48m58.197937329s) [noise=true , address=145.79.227.10:4403 , piece_id=BLZ3NXESHVX6ZSDK2XIBJZZVVQRFP2JPTKAV3OP7ALYALCQ7EGIQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504498154 (33h48m58.200346907s) [noise=true , address=145.79.227.25:4618 , piece_id=WGLEFHBJXXHVVG4CXSND4NY7VQSQ54RIVW5URH25M23BN3OY4UTA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.504943001 (33h48m58.200791754s) [noise=true , address=145.79.227.30:4761 , piece_id=B5PWI34JVTYGOAOYKT6PGCLVEDSDREBZDIFVOAY2AJM3LA4Z6NBQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.514936380 (33h48m58.210785123s) [noise=true , address=145.79.227.31:4799 , piece_id=J3GAOBUTWMSY2YUZAJQWEAKCQ3STOD3O4RQL55AT5OYE2YYEIPYQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.524132682 (33h48m58.219981425s) [noise=true , address=145.79.227.30:4763 , piece_id=NW7IUEK7LZFC2FKRMHSZBNG62HDYYBTZSP6DWE7VA5MWR5HGZQEQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.552496592 (33h48m58.248345345s) [noise=true , address=145.79.227.11:4730 , piece_id=ULDJV6YH6AGWQFSCVMLY4MRI4IHDCHOZBCNQMY6IGOPAPPO5KXVQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.559297612 (33h48m58.255146355s) [noise=true , address=145.79.227.30:4762 , piece_id=BC4KZAX6W543R2W2KPTJUBH5N6YMCOWN5XNLWWGX4MFNMCP26TWQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:26.610754315 (33h48m58.306603068s) [noise=true , address=145.79.227.23:4546 , piece_id=DNHBXHYMCH4EF6NVS5GTRWDEWZGBCBF24MMHZI2AEI5UIRF6JAWA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:28.630741351 (33h49m0.326590094s) [noise=true , address=45.91.116.8:4045 , piece_id=MVZYQCFERDB5RLSVO7RLZE22OTQ6ZYZWDONA3GOVUIUG6BDOVOMA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:29.745758313 (33h49m1.441607067s) [noise=true , address=45.91.116.4:4939 , piece_id=H6WVORQYH5ZO3NLSZXMJC3QZOGAE62N7EQPC4O3DFFOHKTNXPK5A, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778077010 (33h49m3.473925753s) [noise=true , address=45.91.116.9:4263 , piece_id=WLW5K2UIAI72BAQ2D7DSP6ZSXZSG22ZQ47YAUC6PQGKEOLYWJWHQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: <nil>
Oct 31 21:29:31.778098301 (33h49m3.473947044s) [seg_pos={0 0}, upload=29667]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:29:31.778378034 (33h49m3.474226777s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46512->45.91.116.4:4938: use of closed network connection
Oct 31 21:29:31.778385207 (33h49m3.47423396s) [noise=true , address=45.91.116.4:4938 , piece_id=Y7LNMUUH5C4IJ7IOJI2RXOMN4XCNCETONH4KPNRPGNSFB5PUDABQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778547276 (33h49m3.474396019s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59632->77.90.31.18:4807: use of closed network connection
Oct 31 21:29:31.778565610 (33h49m3.474414353s) [noise=true , address=77.90.31.18:4807 , piece_id=YGIASREUD2XVA5NLZT3TLTAJ36MSSJ6R7LHFKQTFHCI5ALLDTSYA, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778607982 (33h49m3.474456735s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57752->77.90.31.15:4697: use of closed network connection
Oct 31 21:29:31.778616498 (33h49m3.474465241s) [noise=true , address=77.90.31.15:4697 , piece_id=KBSSG5VHPEWXVEPDTW5DIQWEVL5FXPWSSGRF3WQ42L7EAWCZECUQ, seg_pos={0 0}, upload=29667]: Long tail cancellation due to optimal threshold
Oct 31 21:29:31.778736786 (33h49m3.47458553s) [seg_pos={0 0}, upload=29667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:29:31.778811989 (33h49m3.474660733s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:29:31.778817441 (33h49m3.474666194s) [upload=29667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:29:32.073519418 (33h49m3.769368161s) [upload=29667]: Done uploading segments...
Oct 31 21:29:32.073537032 (33h49m3.769385785s) [upload=29667]: Upload finished. err: <nil>
Duration: 6.018208926s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................X......XX..................] duration:
5.87664069s trace:
https://jaeger.dev.storj.io/trace/15db8dc6769ded56
FAIL (2025-10-31 21:41:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, Piece ID: QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA) context deadline exceeded; piecestore: (Node ID: 12gTnr8AifLba6znexcCygm4oEkkedFeEHK5ztQKNH8mdiGH6E2, Piece ID: DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA) context deadline exceeded; piecestore: (Node ID: 12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd, Piece ID: NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA) context deadline exceeded
======LOGS======
:28:08.752495154 (33h47m40.448343907s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752742465 (33h47m40.448591218s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.752972904 (33h47m40.448821657s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753165069 (33h47m40.449013823s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753459511 (33h47m40.449308264s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753588355 (33h47m40.449437108s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.753952189 (33h47m40.449800942s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754072187 (33h47m40.44992094s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754473361 (33h47m40.450322114s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754583501 (33h47m40.450432274s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.754935972 (33h47m40.450784725s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755043867 (33h47m40.45089262s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755470099 (33h47m40.451318852s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:08.755667855 (33h47m40.451516638s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Uploading piece...
Oct 31 21:28:10.222276304 (33h47m41.918125057s) [upload=29652]: Next returned nil segment
Oct 31 21:28:10.222296301 (33h47m41.918145055s) [upload=29652]: Waiting for error group managing segments...
Oct 31 21:28:10.624991519 (33h47m42.320840292s) [noise=true , address=145.79.227.33:4894 , piece_id=CBOSFKYQ56EEPQ54L4QMHJE75NTQDYA5XTNVSFJDR2J5WYUC7SRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.637466053 (33h47m42.333314796s) [noise=true , address=145.79.227.21:4494 , piece_id=YFBMQ6LC7DABLI6IGBYV5FSZ7N3TVIW63N4GOUY2JIQJ3PAU6X7A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.714646799 (33h47m42.410495552s) [noise=true , address=145.79.227.38:4992 , piece_id=AISPKACEX3T5OA3JJTIG3SK7CHSUU6Y2X5Q23BNLBNIGST77WHEA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:10.749604966 (33h47m42.445453719s) [noise=true , address=145.79.227.7:4298 , piece_id=OXH67XC6OJFKJTFGXQLBQ6Z25W4M7QSNQ7VDMLBLBWKVQ7LTFHMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:11.600347450 (33h47m43.296196203s) [noise=true , address=145.79.227.16:4921 , piece_id=CBLMEVP2LKL5GXIGPYO42NVIEQG7ZFDB5ZFGQHTQBWR2MHEUK76Q, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.126672165 (33h47m46.822520908s) [noise=true , address=77.90.31.10:4526 , piece_id=M5H3OEJTKM6USZKX26FAYUXOAKDXY2VHMFO25G5ADKRXQJXMVP3A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.244883821 (33h47m46.940732564s) [noise=true , address=77.90.31.15:4729 , piece_id=IALOWQGHQ6M4MUVG6KJUM6ELEHRLUFWPE3SW5PXN23VRLQ4GLAXA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.687902612 (33h47m47.383751365s) [noise=true , address=77.90.31.19:4857 , piece_id=JKBO4LMXFUVS5ORYKFMABB5GGVYOM33T3BHFK5RZDDOUEPH2MNMA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:15.824458331 (33h47m47.520307074s) [noise=true , address=45.91.116.6:4987 , piece_id=QZN6SRAD4GIIAXBLEYFKQL374YPRDHCK67P7ZXI7VU4T35FRUPWA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:16.913580012 (33h47m48.609428755s) [noise=true , address=77.90.31.19:4861 , piece_id=JLZZDP6FMCEXM53YJR37BPYEHTFCIHXKQHAXCRPJ2FEKHKOP4WGA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:20.297114095 (33h47m51.992962838s) [noise=true , address=45.91.116.10:4113 , piece_id=NYEY3WSTWUR3YPY3T4CCHMBGZNO22JFBAUEN7V5VSHOIMHJFZBQA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:23.936121219 (33h47m55.631969972s) [noise=true , address=77.90.31.13:4644 , piece_id=RZ2URVU5WWW3D6T4YLWOGVVTKJ5EFISXLJAPVOKVMYFYFOKI53WQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:26.790564918 (33h47m58.486413661s) [noise=true , address=77.90.31.12:4608 , piece_id=Q4NDDNKL6GCCQVZLU2VIPRXCY4VBRQ7CCF5WHYCJRF5DTG7VDO4A, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803559676 (33h48m6.499408419s) [noise=true , address=45.91.116.7:4026 , piece_id=DJRCLNUZ6CC2DGZDIKLLXY5PFW7BLHT7BRDXCPIURLHASPAOTZYA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: <nil>
Oct 31 21:28:34.803590224 (33h48m6.499438977s) [seg_pos={0 0}, upload=29652]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.803882220 (33h48m6.499730963s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55542->77.90.31.11:4575: use of closed network connection
Oct 31 21:28:34.803898762 (33h48m6.499747505s) [noise=true , address=77.90.31.11:4575 , piece_id=OCONUKIZKNUYWFCRADWDRRXIVAGJV2NJSNEVM7RADQAB36OIMBRA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.803913209 (33h48m6.499761952s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51088->77.90.31.16:4767: use of closed network connection
Oct 31 21:28:34.803933969 (33h48m6.499782712s) [noise=true , address=77.90.31.16:4767 , piece_id=3WVUX32KEQM6SICFFVYMREU6XVITB3YAZUTB4S4WQSJIMZWLD4PA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804027547 (33h48m6.4998763s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49862->77.90.31.14:4692: use of closed network connection
Oct 31 21:28:34.804038367 (33h48m6.49988711s) [noise=true , address=77.90.31.14:4692 , piece_id=U65MYLUS5KGRSK4PP2I63QTGYRZZICJZJFPTIGOLLUTSJCYHNLXQ, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804234060 (33h48m6.500082813s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:34.804243187 (33h48m6.500091941s) [noise=true , address=77.90.31.17:4774 , piece_id=DOPPNREJEPJ4JOKG5PUBPAWEZDUJ7A7ZZ4SYLS2NNBD44OLLXMAA, seg_pos={0 0}, upload=29652]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.804264467 (33h48m6.50011321s) [seg_pos={0 0}, upload=29652]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.804401328 (33h48m6.500250081s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.804408943 (33h48m6.500257696s) [upload=29652]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:35.103814398 (33h48m6.799663151s) [upload=29652]: Done uploading segments...
Oct 31 21:28:35.103837311 (33h48m6.799686054s) [upload=29652]: Upload finished. err: <nil>
Duration: 5.876670908s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [................................XX......X...................] duration:
3.48147449s trace:
https://jaeger.dev.storj.io/trace/d551b29a119bb
FAIL (2025-10-31 21:41:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.481512774s
core
uplink
us-select-1
us1-placement-20
|