Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..............X.X.X.............X..........................] duration: 36.798103675s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:14:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 36.798131077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X..XX..X...X.XXXX.X........X...X............XX....X.X..X..] duration: 5.667199535s trace: https://jaeger.dev.storj.io/trace/5898fa4f3c32d4f9 FAIL (2025-12-29 08:13:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: BYIPHL5SZ7JKW6H5OI6TJZBCZU6NUBGO7MEQCE7NPV5K5XAWNISA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: GFBWBPOWAVR7EHTN2A7Q6HOE4Y4NDJ5UFDFOF77CTOQEJSJM2ZRQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: XY56W4Q76OGOJ5DV5SO65AARLEN7DAZFVRGSAUVWPVUW32Y2P55A) context deadline exceeded ======LOGS====== (568h43m40.720086885s) [upload=498609]: Done starting upload Dec 29 08:12:40.849109341 (568h43m40.720192376s) [upload=498609]: Uploading segments... Dec 29 08:12:40.849249519 (568h43m40.720332554s) [upload=498609]: Got next segment. Inline: false Dec 29 08:12:40.849279725 (568h43m40.72036277s) [upload=498609]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:12:40.849284916 (568h43m40.720367951s) [upload=498609]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:12:41.096055260 (568h43m40.967138305s) [seg_pos={0 0}, upload=498609]: Begin upload segment... Dec 29 08:12:41.096126847 (568h43m40.967209882s) [seg_pos={0 0}, upload=498609]: Done begin upload segment. Dec 29 08:12:41.096182512 (568h43m40.967265547s) [noise=true , address=145.79.227.52:4445 , piece_id=B4HLA5YGGGHUZZR7CL3ARC4MSEMRTTT4O2C2AXUXUU6G2YZ6HLJQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.096215876 (568h43m40.967298921s) [noise=true , address=45.91.116.7:4010 , piece_id=GFBWBPOWAVR7EHTN2A7Q6HOE4Y4NDJ5UFDFOF77CTOQEJSJM2ZRQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.096791853 (568h43m40.967874888s) [noise=true , address=145.79.227.18:4007 , piece_id=Y6ZR3XGCS7KNUTD3DLDBU6B4MFA347BB6NHYUBPEOHE6FJYVN5NQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.097084610 (568h43m40.968167655s) [noise=true , address=145.79.227.50:4375 , piece_id=XY56W4Q76OGOJ5DV5SO65AARLEN7DAZFVRGSAUVWPVUW32Y2P55A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.097366336 (568h43m40.968449371s) [noise=true , address=145.79.227.38:4977 , piece_id=RFPY5SGIBQNE54NVNLQXNA6IVVSCOA6OUYE66237FYP3SN6XRD4A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.097713527 (568h43m40.968796572s) [noise=true , address=45.91.116.10:4083 , piece_id=GQVHWKBE4HILQDEOWDHPBQ7HWP6KLGXDDN354KYZIP5TEKPK57NA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.099664282 (568h43m40.970747327s) [noise=true , address=45.91.116.8:4045 , piece_id=PLTIDL4IORN7OTXDYT6RC24LSLUPVATD3ODN3YXQFDHLSGDGSN5A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.101006638 (568h43m40.972089673s) [noise=true , address=145.79.227.43:4158 , piece_id=DEZQHGQBYYJAOAJ2D35VOTKDCXOO3XCIZVZCK5JGEFBCEYVR3QYA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.102390253 (568h43m40.973473288s) [noise=true , address=45.91.116.11:4117 , piece_id=BIPTVR3YJXBFF53MYNP55SO35EGHL3FLN26CVTZOIPVDKV2FVMZQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.103668186 (568h43m40.974751221s) [noise=true , address=45.91.116.4:4937 , piece_id=3JTVSLPQGTO3CK4KJ3VOKNXD53RKIAZG2OSAGRLYIHIEFOJZDEBA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.104765917 (568h43m40.975848952s) [noise=true , address=145.79.227.17:4972 , piece_id=JOGXC72F76UY7GGV7GAZZUHDUS6AUUK6QMTNWDZK573DSQOSNS5Q, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.105892331 (568h43m40.976975687s) [noise=true , address=45.91.116.12:4225 , piece_id=BYIPHL5SZ7JKW6H5OI6TJZBCZU6NUBGO7MEQCE7NPV5K5XAWNISA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.106454712 (568h43m40.977537757s) [noise=true , address=145.79.227.31:4798 , piece_id=DR5F74A2KV6MWG5ZSVZL2CCHZDNY54CAFPYYV2EV6Z7V7O7BBPLA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.108262655 (568h43m40.97934569s) [noise=true , address=45.91.116.6:4974 , piece_id=56AZDEX32QATW66AHEF4TP6GXJHS4IRWJ2MJSSNW67WV7M33NVQQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.108889498 (568h43m40.979972543s) [noise=true , address=145.79.227.19:4079 , piece_id=CYLXFOKFT6GHVSEHBYM7QH5K2UTXYITWYSQTD4SILYTQEMIGWG5A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.111670845 (568h43m40.98275388s) [noise=true , address=145.79.227.39:4013 , piece_id=XWVXCUSL4Y5ZGJENOAGWMGRXVLPRLXA5BIVXUWJB2G2RJUUSIOCQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:42.696794178 (568h43m42.567877213s) [upload=498609]: Next returned nil segment Dec 29 08:12:42.696808726 (568h43m42.567891761s) [upload=498609]: Waiting for error group managing segments... Dec 29 08:12:43.079849459 (568h43m42.950932494s) [noise=true , address=145.79.227.18:4007 , piece_id=Y6ZR3XGCS7KNUTD3DLDBU6B4MFA347BB6NHYUBPEOHE6FJYVN5NQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.080068586 (568h43m42.951151631s) [noise=true , address=145.79.227.52:4445 , piece_id=B4HLA5YGGGHUZZR7CL3ARC4MSEMRTTT4O2C2AXUXUU6G2YZ6HLJQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.086973985 (568h43m42.95805702s) [noise=true , address=145.79.227.38:4977 , piece_id=RFPY5SGIBQNE54NVNLQXNA6IVVSCOA6OUYE66237FYP3SN6XRD4A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.101082890 (568h43m42.972165935s) [noise=true , address=145.79.227.31:4798 , piece_id=DR5F74A2KV6MWG5ZSVZL2CCHZDNY54CAFPYYV2EV6Z7V7O7BBPLA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.101272020 (568h43m42.972355055s) [noise=true , address=145.79.227.19:4079 , piece_id=CYLXFOKFT6GHVSEHBYM7QH5K2UTXYITWYSQTD4SILYTQEMIGWG5A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.106875740 (568h43m42.977958786s) [noise=true , address=145.79.227.39:4013 , piece_id=XWVXCUSL4Y5ZGJENOAGWMGRXVLPRLXA5BIVXUWJB2G2RJUUSIOCQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.122101212 (568h43m42.993184267s) [noise=true , address=145.79.227.50:4375 , piece_id=XY56W4Q76OGOJ5DV5SO65AARLEN7DAZFVRGSAUVWPVUW32Y2P55A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.129679371 (568h43m43.000762406s) [noise=true , address=145.79.227.17:4972 , piece_id=JOGXC72F76UY7GGV7GAZZUHDUS6AUUK6QMTNWDZK573DSQOSNS5Q, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.336488134 (568h43m43.207571169s) [noise=true , address=45.91.116.10:4083 , piece_id=GQVHWKBE4HILQDEOWDHPBQ7HWP6KLGXDDN354KYZIP5TEKPK57NA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.459293572 (568h43m43.330376607s) [noise=true , address=45.91.116.12:4225 , piece_id=BYIPHL5SZ7JKW6H5OI6TJZBCZU6NUBGO7MEQCE7NPV5K5XAWNISA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.492015650 (568h43m43.363098685s) [noise=true , address=45.91.116.6:4974 , piece_id=56AZDEX32QATW66AHEF4TP6GXJHS4IRWJ2MJSSNW67WV7M33NVQQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:44.051981422 (568h43m43.923064467s) [noise=true , address=45.91.116.4:4937 , piece_id=3JTVSLPQGTO3CK4KJ3VOKNXD53RKIAZG2OSAGRLYIHIEFOJZDEBA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:44.063190305 (568h43m43.93427335s) [noise=true , address=45.91.116.7:4010 , piece_id=GFBWBPOWAVR7EHTN2A7Q6HOE4Y4NDJ5UFDFOF77CTOQEJSJM2ZRQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:44.063216855 (568h43m43.934299891s) [seg_pos={0 0}, upload=498609]: Segment reached optimal threshold of 13 pieces. Dec 29 08:12:44.063471020 (568h43m43.934554085s) [noise=true , address=45.91.116.11:4117 , piece_id=BIPTVR3YJXBFF53MYNP55SO35EGHL3FLN26CVTZOIPVDKV2FVMZQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:59122->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:59122->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59122->45.91.116.11:4117: use of closed network connection Dec 29 08:12:44.063481700 (568h43m43.934564745s) [noise=true , address=45.91.116.11:4117 , piece_id=BIPTVR3YJXBFF53MYNP55SO35EGHL3FLN26CVTZOIPVDKV2FVMZQ, seg_pos={0 0}, upload=498609]: Long tail cancellation due to optimal threshold Dec 29 08:12:44.063929874 (568h43m43.935012909s) [noise=true , address=145.79.227.43:4158 , piece_id=DEZQHGQBYYJAOAJ2D35VOTKDCXOO3XCIZVZCK5JGEFBCEYVR3QYA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 08:12:44.063949611 (568h43m43.935032646s) [noise=true , address=145.79.227.43:4158 , piece_id=DEZQHGQBYYJAOAJ2D35VOTKDCXOO3XCIZVZCK5JGEFBCEYVR3QYA, seg_pos={0 0}, upload=498609]: Long tail cancellation due to optimal threshold Dec 29 08:12:44.063967766 (568h43m43.935050811s) [noise=true , address=45.91.116.8:4045 , piece_id=PLTIDL4IORN7OTXDYT6RC24LSLUPVATD3ODN3YXQFDHLSGDGSN5A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:58604->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 08:12:44.063980139 (568h43m43.935063174s) [noise=true , address=45.91.116.8:4045 , piece_id=PLTIDL4IORN7OTXDYT6RC24LSLUPVATD3ODN3YXQFDHLSGDGSN5A, seg_pos={0 0}, upload=498609]: Long tail cancellation due to optimal threshold Dec 29 08:12:44.064005297 (568h43m43.935088342s) [seg_pos={0 0}, upload=498609]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:12:44.064117461 (568h43m43.935200506s) [upload=498609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:12:44.064125405 (568h43m43.935208451s) [upload=498609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:12:44.395912524 (568h43m44.266995559s) [upload=498609]: Done uploading segments... Dec 29 08:12:44.395933434 (568h43m44.26701647s) [upload=498609]: Upload finished. err: <nil> Duration: 5.667231375s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..............X.X.X.............X.........................] duration: 32.726579535s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:13:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 32.726614993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X..XX..X...X.XXXX.X........X...X............XX....X.X..X.] duration: 5.815671873s trace: https://jaeger.dev.storj.io/trace/5898fa4f3c32d4f9 FAIL (2025-12-29 08:13:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: BYIPHL5SZ7JKW6H5OI6TJZBCZU6NUBGO7MEQCE7NPV5K5XAWNISA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: GFBWBPOWAVR7EHTN2A7Q6HOE4Y4NDJ5UFDFOF77CTOQEJSJM2ZRQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: XY56W4Q76OGOJ5DV5SO65AARLEN7DAZFVRGSAUVWPVUW32Y2P55A) context deadline exceeded ======LOGS====== (568h43m40.720086885s) [upload=498609]: Done starting upload Dec 29 08:12:40.849109341 (568h43m40.720192376s) [upload=498609]: Uploading segments... Dec 29 08:12:40.849249519 (568h43m40.720332554s) [upload=498609]: Got next segment. Inline: false Dec 29 08:12:40.849279725 (568h43m40.72036277s) [upload=498609]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:12:40.849284916 (568h43m40.720367951s) [upload=498609]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:12:41.096055260 (568h43m40.967138305s) [seg_pos={0 0}, upload=498609]: Begin upload segment... Dec 29 08:12:41.096126847 (568h43m40.967209882s) [seg_pos={0 0}, upload=498609]: Done begin upload segment. Dec 29 08:12:41.096182512 (568h43m40.967265547s) [noise=true , address=145.79.227.52:4445 , piece_id=B4HLA5YGGGHUZZR7CL3ARC4MSEMRTTT4O2C2AXUXUU6G2YZ6HLJQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.096215876 (568h43m40.967298921s) [noise=true , address=45.91.116.7:4010 , piece_id=GFBWBPOWAVR7EHTN2A7Q6HOE4Y4NDJ5UFDFOF77CTOQEJSJM2ZRQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.096791853 (568h43m40.967874888s) [noise=true , address=145.79.227.18:4007 , piece_id=Y6ZR3XGCS7KNUTD3DLDBU6B4MFA347BB6NHYUBPEOHE6FJYVN5NQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.097084610 (568h43m40.968167655s) [noise=true , address=145.79.227.50:4375 , piece_id=XY56W4Q76OGOJ5DV5SO65AARLEN7DAZFVRGSAUVWPVUW32Y2P55A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.097366336 (568h43m40.968449371s) [noise=true , address=145.79.227.38:4977 , piece_id=RFPY5SGIBQNE54NVNLQXNA6IVVSCOA6OUYE66237FYP3SN6XRD4A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.097713527 (568h43m40.968796572s) [noise=true , address=45.91.116.10:4083 , piece_id=GQVHWKBE4HILQDEOWDHPBQ7HWP6KLGXDDN354KYZIP5TEKPK57NA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.099664282 (568h43m40.970747327s) [noise=true , address=45.91.116.8:4045 , piece_id=PLTIDL4IORN7OTXDYT6RC24LSLUPVATD3ODN3YXQFDHLSGDGSN5A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.101006638 (568h43m40.972089673s) [noise=true , address=145.79.227.43:4158 , piece_id=DEZQHGQBYYJAOAJ2D35VOTKDCXOO3XCIZVZCK5JGEFBCEYVR3QYA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.102390253 (568h43m40.973473288s) [noise=true , address=45.91.116.11:4117 , piece_id=BIPTVR3YJXBFF53MYNP55SO35EGHL3FLN26CVTZOIPVDKV2FVMZQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.103668186 (568h43m40.974751221s) [noise=true , address=45.91.116.4:4937 , piece_id=3JTVSLPQGTO3CK4KJ3VOKNXD53RKIAZG2OSAGRLYIHIEFOJZDEBA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.104765917 (568h43m40.975848952s) [noise=true , address=145.79.227.17:4972 , piece_id=JOGXC72F76UY7GGV7GAZZUHDUS6AUUK6QMTNWDZK573DSQOSNS5Q, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.105892331 (568h43m40.976975687s) [noise=true , address=45.91.116.12:4225 , piece_id=BYIPHL5SZ7JKW6H5OI6TJZBCZU6NUBGO7MEQCE7NPV5K5XAWNISA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.106454712 (568h43m40.977537757s) [noise=true , address=145.79.227.31:4798 , piece_id=DR5F74A2KV6MWG5ZSVZL2CCHZDNY54CAFPYYV2EV6Z7V7O7BBPLA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.108262655 (568h43m40.97934569s) [noise=true , address=45.91.116.6:4974 , piece_id=56AZDEX32QATW66AHEF4TP6GXJHS4IRWJ2MJSSNW67WV7M33NVQQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.108889498 (568h43m40.979972543s) [noise=true , address=145.79.227.19:4079 , piece_id=CYLXFOKFT6GHVSEHBYM7QH5K2UTXYITWYSQTD4SILYTQEMIGWG5A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.111670845 (568h43m40.98275388s) [noise=true , address=145.79.227.39:4013 , piece_id=XWVXCUSL4Y5ZGJENOAGWMGRXVLPRLXA5BIVXUWJB2G2RJUUSIOCQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:42.696794178 (568h43m42.567877213s) [upload=498609]: Next returned nil segment Dec 29 08:12:42.696808726 (568h43m42.567891761s) [upload=498609]: Waiting for error group managing segments... Dec 29 08:12:43.079849459 (568h43m42.950932494s) [noise=true , address=145.79.227.18:4007 , piece_id=Y6ZR3XGCS7KNUTD3DLDBU6B4MFA347BB6NHYUBPEOHE6FJYVN5NQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.080068586 (568h43m42.951151631s) [noise=true , address=145.79.227.52:4445 , piece_id=B4HLA5YGGGHUZZR7CL3ARC4MSEMRTTT4O2C2AXUXUU6G2YZ6HLJQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.086973985 (568h43m42.95805702s) [noise=true , address=145.79.227.38:4977 , piece_id=RFPY5SGIBQNE54NVNLQXNA6IVVSCOA6OUYE66237FYP3SN6XRD4A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.101082890 (568h43m42.972165935s) [noise=true , address=145.79.227.31:4798 , piece_id=DR5F74A2KV6MWG5ZSVZL2CCHZDNY54CAFPYYV2EV6Z7V7O7BBPLA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.101272020 (568h43m42.972355055s) [noise=true , address=145.79.227.19:4079 , piece_id=CYLXFOKFT6GHVSEHBYM7QH5K2UTXYITWYSQTD4SILYTQEMIGWG5A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.106875740 (568h43m42.977958786s) [noise=true , address=145.79.227.39:4013 , piece_id=XWVXCUSL4Y5ZGJENOAGWMGRXVLPRLXA5BIVXUWJB2G2RJUUSIOCQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.122101212 (568h43m42.993184267s) [noise=true , address=145.79.227.50:4375 , piece_id=XY56W4Q76OGOJ5DV5SO65AARLEN7DAZFVRGSAUVWPVUW32Y2P55A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.129679371 (568h43m43.000762406s) [noise=true , address=145.79.227.17:4972 , piece_id=JOGXC72F76UY7GGV7GAZZUHDUS6AUUK6QMTNWDZK573DSQOSNS5Q, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.336488134 (568h43m43.207571169s) [noise=true , address=45.91.116.10:4083 , piece_id=GQVHWKBE4HILQDEOWDHPBQ7HWP6KLGXDDN354KYZIP5TEKPK57NA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.459293572 (568h43m43.330376607s) [noise=true , address=45.91.116.12:4225 , piece_id=BYIPHL5SZ7JKW6H5OI6TJZBCZU6NUBGO7MEQCE7NPV5K5XAWNISA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.492015650 (568h43m43.363098685s) [noise=true , address=45.91.116.6:4974 , piece_id=56AZDEX32QATW66AHEF4TP6GXJHS4IRWJ2MJSSNW67WV7M33NVQQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:44.051981422 (568h43m43.923064467s) [noise=true , address=45.91.116.4:4937 , piece_id=3JTVSLPQGTO3CK4KJ3VOKNXD53RKIAZG2OSAGRLYIHIEFOJZDEBA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:44.063190305 (568h43m43.93427335s) [noise=true , address=45.91.116.7:4010 , piece_id=GFBWBPOWAVR7EHTN2A7Q6HOE4Y4NDJ5UFDFOF77CTOQEJSJM2ZRQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:44.063216855 (568h43m43.934299891s) [seg_pos={0 0}, upload=498609]: Segment reached optimal threshold of 13 pieces. Dec 29 08:12:44.063471020 (568h43m43.934554085s) [noise=true , address=45.91.116.11:4117 , piece_id=BIPTVR3YJXBFF53MYNP55SO35EGHL3FLN26CVTZOIPVDKV2FVMZQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:59122->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:59122->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59122->45.91.116.11:4117: use of closed network connection Dec 29 08:12:44.063481700 (568h43m43.934564745s) [noise=true , address=45.91.116.11:4117 , piece_id=BIPTVR3YJXBFF53MYNP55SO35EGHL3FLN26CVTZOIPVDKV2FVMZQ, seg_pos={0 0}, upload=498609]: Long tail cancellation due to optimal threshold Dec 29 08:12:44.063929874 (568h43m43.935012909s) [noise=true , address=145.79.227.43:4158 , piece_id=DEZQHGQBYYJAOAJ2D35VOTKDCXOO3XCIZVZCK5JGEFBCEYVR3QYA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 08:12:44.063949611 (568h43m43.935032646s) [noise=true , address=145.79.227.43:4158 , piece_id=DEZQHGQBYYJAOAJ2D35VOTKDCXOO3XCIZVZCK5JGEFBCEYVR3QYA, seg_pos={0 0}, upload=498609]: Long tail cancellation due to optimal threshold Dec 29 08:12:44.063967766 (568h43m43.935050811s) [noise=true , address=45.91.116.8:4045 , piece_id=PLTIDL4IORN7OTXDYT6RC24LSLUPVATD3ODN3YXQFDHLSGDGSN5A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:58604->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 08:12:44.063980139 (568h43m43.935063174s) [noise=true , address=45.91.116.8:4045 , piece_id=PLTIDL4IORN7OTXDYT6RC24LSLUPVATD3ODN3YXQFDHLSGDGSN5A, seg_pos={0 0}, upload=498609]: Long tail cancellation due to optimal threshold Dec 29 08:12:44.064005297 (568h43m43.935088342s) [seg_pos={0 0}, upload=498609]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:12:44.064117461 (568h43m43.935200506s) [upload=498609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:12:44.064125405 (568h43m43.935208451s) [upload=498609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:12:44.395912524 (568h43m44.266995559s) [upload=498609]: Done uploading segments... Dec 29 08:12:44.395933434 (568h43m44.26701647s) [upload=498609]: Upload finished. err: <nil> Duration: 5.815688144s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................X........................X..X......X........] duration: 3.490015884s trace: https://jaeger.dev.storj.io/trace/43163810b1d7c614 FAIL (2025-12-29 08:13:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49004516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...X..XX..X...X.XXXX.X........X...X............XX....X.X..X] duration: 30.00356642s trace: https://jaeger.dev.storj.io/trace/5898fa4f3c32d4f9 FAIL (2025-12-29 08:13:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: BYIPHL5SZ7JKW6H5OI6TJZBCZU6NUBGO7MEQCE7NPV5K5XAWNISA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: GFBWBPOWAVR7EHTN2A7Q6HOE4Y4NDJ5UFDFOF77CTOQEJSJM2ZRQ) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: XY56W4Q76OGOJ5DV5SO65AARLEN7DAZFVRGSAUVWPVUW32Y2P55A) context deadline exceeded ======LOGS====== (568h43m40.720086885s) [upload=498609]: Done starting upload Dec 29 08:12:40.849109341 (568h43m40.720192376s) [upload=498609]: Uploading segments... Dec 29 08:12:40.849249519 (568h43m40.720332554s) [upload=498609]: Got next segment. Inline: false Dec 29 08:12:40.849279725 (568h43m40.72036277s) [upload=498609]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:12:40.849284916 (568h43m40.720367951s) [upload=498609]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:12:41.096055260 (568h43m40.967138305s) [seg_pos={0 0}, upload=498609]: Begin upload segment... Dec 29 08:12:41.096126847 (568h43m40.967209882s) [seg_pos={0 0}, upload=498609]: Done begin upload segment. Dec 29 08:12:41.096182512 (568h43m40.967265547s) [noise=true , address=145.79.227.52:4445 , piece_id=B4HLA5YGGGHUZZR7CL3ARC4MSEMRTTT4O2C2AXUXUU6G2YZ6HLJQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.096215876 (568h43m40.967298921s) [noise=true , address=45.91.116.7:4010 , piece_id=GFBWBPOWAVR7EHTN2A7Q6HOE4Y4NDJ5UFDFOF77CTOQEJSJM2ZRQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.096791853 (568h43m40.967874888s) [noise=true , address=145.79.227.18:4007 , piece_id=Y6ZR3XGCS7KNUTD3DLDBU6B4MFA347BB6NHYUBPEOHE6FJYVN5NQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.097084610 (568h43m40.968167655s) [noise=true , address=145.79.227.50:4375 , piece_id=XY56W4Q76OGOJ5DV5SO65AARLEN7DAZFVRGSAUVWPVUW32Y2P55A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.097366336 (568h43m40.968449371s) [noise=true , address=145.79.227.38:4977 , piece_id=RFPY5SGIBQNE54NVNLQXNA6IVVSCOA6OUYE66237FYP3SN6XRD4A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.097713527 (568h43m40.968796572s) [noise=true , address=45.91.116.10:4083 , piece_id=GQVHWKBE4HILQDEOWDHPBQ7HWP6KLGXDDN354KYZIP5TEKPK57NA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.099664282 (568h43m40.970747327s) [noise=true , address=45.91.116.8:4045 , piece_id=PLTIDL4IORN7OTXDYT6RC24LSLUPVATD3ODN3YXQFDHLSGDGSN5A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.101006638 (568h43m40.972089673s) [noise=true , address=145.79.227.43:4158 , piece_id=DEZQHGQBYYJAOAJ2D35VOTKDCXOO3XCIZVZCK5JGEFBCEYVR3QYA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.102390253 (568h43m40.973473288s) [noise=true , address=45.91.116.11:4117 , piece_id=BIPTVR3YJXBFF53MYNP55SO35EGHL3FLN26CVTZOIPVDKV2FVMZQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.103668186 (568h43m40.974751221s) [noise=true , address=45.91.116.4:4937 , piece_id=3JTVSLPQGTO3CK4KJ3VOKNXD53RKIAZG2OSAGRLYIHIEFOJZDEBA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.104765917 (568h43m40.975848952s) [noise=true , address=145.79.227.17:4972 , piece_id=JOGXC72F76UY7GGV7GAZZUHDUS6AUUK6QMTNWDZK573DSQOSNS5Q, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.105892331 (568h43m40.976975687s) [noise=true , address=45.91.116.12:4225 , piece_id=BYIPHL5SZ7JKW6H5OI6TJZBCZU6NUBGO7MEQCE7NPV5K5XAWNISA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.106454712 (568h43m40.977537757s) [noise=true , address=145.79.227.31:4798 , piece_id=DR5F74A2KV6MWG5ZSVZL2CCHZDNY54CAFPYYV2EV6Z7V7O7BBPLA, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.108262655 (568h43m40.97934569s) [noise=true , address=45.91.116.6:4974 , piece_id=56AZDEX32QATW66AHEF4TP6GXJHS4IRWJ2MJSSNW67WV7M33NVQQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.108889498 (568h43m40.979972543s) [noise=true , address=145.79.227.19:4079 , piece_id=CYLXFOKFT6GHVSEHBYM7QH5K2UTXYITWYSQTD4SILYTQEMIGWG5A, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:41.111670845 (568h43m40.98275388s) [noise=true , address=145.79.227.39:4013 , piece_id=XWVXCUSL4Y5ZGJENOAGWMGRXVLPRLXA5BIVXUWJB2G2RJUUSIOCQ, seg_pos={0 0}, upload=498609]: Uploading piece... Dec 29 08:12:42.696794178 (568h43m42.567877213s) [upload=498609]: Next returned nil segment Dec 29 08:12:42.696808726 (568h43m42.567891761s) [upload=498609]: Waiting for error group managing segments... Dec 29 08:12:43.079849459 (568h43m42.950932494s) [noise=true , address=145.79.227.18:4007 , piece_id=Y6ZR3XGCS7KNUTD3DLDBU6B4MFA347BB6NHYUBPEOHE6FJYVN5NQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.080068586 (568h43m42.951151631s) [noise=true , address=145.79.227.52:4445 , piece_id=B4HLA5YGGGHUZZR7CL3ARC4MSEMRTTT4O2C2AXUXUU6G2YZ6HLJQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.086973985 (568h43m42.95805702s) [noise=true , address=145.79.227.38:4977 , piece_id=RFPY5SGIBQNE54NVNLQXNA6IVVSCOA6OUYE66237FYP3SN6XRD4A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.101082890 (568h43m42.972165935s) [noise=true , address=145.79.227.31:4798 , piece_id=DR5F74A2KV6MWG5ZSVZL2CCHZDNY54CAFPYYV2EV6Z7V7O7BBPLA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.101272020 (568h43m42.972355055s) [noise=true , address=145.79.227.19:4079 , piece_id=CYLXFOKFT6GHVSEHBYM7QH5K2UTXYITWYSQTD4SILYTQEMIGWG5A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.106875740 (568h43m42.977958786s) [noise=true , address=145.79.227.39:4013 , piece_id=XWVXCUSL4Y5ZGJENOAGWMGRXVLPRLXA5BIVXUWJB2G2RJUUSIOCQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.122101212 (568h43m42.993184267s) [noise=true , address=145.79.227.50:4375 , piece_id=XY56W4Q76OGOJ5DV5SO65AARLEN7DAZFVRGSAUVWPVUW32Y2P55A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.129679371 (568h43m43.000762406s) [noise=true , address=145.79.227.17:4972 , piece_id=JOGXC72F76UY7GGV7GAZZUHDUS6AUUK6QMTNWDZK573DSQOSNS5Q, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.336488134 (568h43m43.207571169s) [noise=true , address=45.91.116.10:4083 , piece_id=GQVHWKBE4HILQDEOWDHPBQ7HWP6KLGXDDN354KYZIP5TEKPK57NA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.459293572 (568h43m43.330376607s) [noise=true , address=45.91.116.12:4225 , piece_id=BYIPHL5SZ7JKW6H5OI6TJZBCZU6NUBGO7MEQCE7NPV5K5XAWNISA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:43.492015650 (568h43m43.363098685s) [noise=true , address=45.91.116.6:4974 , piece_id=56AZDEX32QATW66AHEF4TP6GXJHS4IRWJ2MJSSNW67WV7M33NVQQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:44.051981422 (568h43m43.923064467s) [noise=true , address=45.91.116.4:4937 , piece_id=3JTVSLPQGTO3CK4KJ3VOKNXD53RKIAZG2OSAGRLYIHIEFOJZDEBA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:44.063190305 (568h43m43.93427335s) [noise=true , address=45.91.116.7:4010 , piece_id=GFBWBPOWAVR7EHTN2A7Q6HOE4Y4NDJ5UFDFOF77CTOQEJSJM2ZRQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: <nil> Dec 29 08:12:44.063216855 (568h43m43.934299891s) [seg_pos={0 0}, upload=498609]: Segment reached optimal threshold of 13 pieces. Dec 29 08:12:44.063471020 (568h43m43.934554085s) [noise=true , address=45.91.116.11:4117 , piece_id=BIPTVR3YJXBFF53MYNP55SO35EGHL3FLN26CVTZOIPVDKV2FVMZQ, seg_pos={0 0}, upload=498609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:59122->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:59122->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59122->45.91.116.11:4117: use of closed network connection Dec 29 08:12:44.063481700 (568h43m43.934564745s) [noise=true , address=45.91.116.11:4117 , piece_id=BIPTVR3YJXBFF53MYNP55SO35EGHL3FLN26CVTZOIPVDKV2FVMZQ, seg_pos={0 0}, upload=498609]: Long tail cancellation due to optimal threshold Dec 29 08:12:44.063929874 (568h43m43.935012909s) [noise=true , address=145.79.227.43:4158 , piece_id=DEZQHGQBYYJAOAJ2D35VOTKDCXOO3XCIZVZCK5JGEFBCEYVR3QYA, seg_pos={0 0}, upload=498609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 08:12:44.063949611 (568h43m43.935032646s) [noise=true , address=145.79.227.43:4158 , piece_id=DEZQHGQBYYJAOAJ2D35VOTKDCXOO3XCIZVZCK5JGEFBCEYVR3QYA, seg_pos={0 0}, upload=498609]: Long tail cancellation due to optimal threshold Dec 29 08:12:44.063967766 (568h43m43.935050811s) [noise=true , address=45.91.116.8:4045 , piece_id=PLTIDL4IORN7OTXDYT6RC24LSLUPVATD3ODN3YXQFDHLSGDGSN5A, seg_pos={0 0}, upload=498609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:58604->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 08:12:44.063980139 (568h43m43.935063174s) [noise=true , address=45.91.116.8:4045 , piece_id=PLTIDL4IORN7OTXDYT6RC24LSLUPVATD3ODN3YXQFDHLSGDGSN5A, seg_pos={0 0}, upload=498609]: Long tail cancellation due to optimal threshold Dec 29 08:12:44.064005297 (568h43m43.935088342s) [seg_pos={0 0}, upload=498609]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:12:44.064117461 (568h43m43.935200506s) [upload=498609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:12:44.064125405 (568h43m43.935208451s) [upload=498609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:12:44.395912524 (568h43m44.266995559s) [upload=498609]: Done uploading segments... Dec 29 08:12:44.395933434 (568h43m44.26701647s) [upload=498609]: Upload finished. err: <nil> Duration: 30.003586829s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................X........................X..X......X.......] duration: 3.35937429s trace: https://jaeger.dev.storj.io/trace/43163810b1d7c614 FAIL (2025-12-29 08:12:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.359399527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..............X.X.X.............X........................] duration: 15.33273423s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:12:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 15.332762904s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................X........................X..X......X......] duration: 3.494558972s trace: https://jaeger.dev.storj.io/trace/43163810b1d7c614 FAIL (2025-12-29 08:12:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494589489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X...X..XX..X...X.XXXX.X........X...X............XX....X.X..] duration: 5.923697668s trace: https://jaeger.dev.storj.io/trace/6a66a6c26a16039b FAIL (2025-12-29 08:11:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: TVILUK5NEJOMZDTEJPPDHNSXO7TSNEIWWVSOZLV6544MM46YSMTA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 7BN4ZP6QKPNXQS2RWYDM5GY773UBS3E73MY2IL2R55SDWYF4B4UQ) context deadline exceeded ======LOGS====== Dec 29 08:09:25.020604922 (568h40m24.891687957s) [upload=498563]: Starting upload Dec 29 08:09:25.020673453 (568h40m24.891756488s) [upload=498563]: Done starting upload Dec 29 08:09:25.020864347 (568h40m24.891947382s) [upload=498563]: Uploading segments... Dec 29 08:09:25.020894374 (568h40m24.891977409s) [upload=498563]: Got next segment. Inline: false Dec 29 08:09:25.020929842 (568h40m24.892012887s) [upload=498563]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:09:25.020935292 (568h40m24.892018327s) [upload=498563]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:09:25.266662981 (568h40m25.137746016s) [seg_pos={0 0}, upload=498563]: Begin upload segment... Dec 29 08:09:25.266701785 (568h40m25.13778482s) [seg_pos={0 0}, upload=498563]: Done begin upload segment. Dec 29 08:09:25.266718437 (568h40m25.137801472s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRG42GDDRJER47YL5H6AOL5VSJ4GSK6CCY4PRRSOHVMZ7VHK3PQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.266727664 (568h40m25.137810709s) [noise=true , address=145.79.227.18:4006 , piece_id=LQJZVYYWL62UD5P52QYG2YD4VKUTSK6UJIRELUOGUJ4CIX4ZPU2Q, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.267179214 (568h40m25.138262249s) [noise=true , address=45.91.116.8:4047 , piece_id=2LSTDDF7PU2QZUCP2NSHI54U2MVZRXCXOOKETMUDYMGKGITLPU6A, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.267840043 (568h40m25.138923078s) [noise=true , address=145.79.227.2:4115 , piece_id=WZMXOUIJIGKLX4GDKNXX3NUZ4PDGN2RNQMQYE6J5UHFLPONG6NFA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268114956 (568h40m25.139197991s) [noise=true , address=77.90.31.18:4805 , piece_id=7BN4ZP6QKPNXQS2RWYDM5GY773UBS3E73MY2IL2R55SDWYF4B4UQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268400049 (568h40m25.139483084s) [noise=true , address=145.79.227.31:4798 , piece_id=I32JWMKL6HDTQAREE3IDJKRVQYMN7GAHAZIFWG3YRKIV4HN3BSFQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268667088 (568h40m25.139750133s) [noise=true , address=77.90.31.16:4735 , piece_id=CGXZ4ZDVW4R2BQZTTGBSW7HPURF2JJRFGWL2VGULRJN2F65LTEPQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268930779 (568h40m25.140013814s) [noise=true , address=145.79.227.42:4121 , piece_id=TZJ27VQFY3J53WAGT5G2WVGSTIEEJYTU3JJOF6WVVHATHSTLL7DA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.269238936 (568h40m25.140321971s) [noise=true , address=45.91.116.9:4262 , piece_id=TVILUK5NEJOMZDTEJPPDHNSXO7TSNEIWWVSOZLV6544MM46YSMTA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.270262556 (568h40m25.141345601s) [noise=true , address=77.90.31.10:4519 , piece_id=OJVPUGJLS4D5OY7W37IGLV4MBOTYLLWA5VJ6FIR37MJO6NXWXE6Q, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.270590490 (568h40m25.141673525s) [noise=true , address=45.91.116.11:4120 , piece_id=LJQV4BLS2ASIVWJE5KEB4OQY3BDHH3DEQ6N2FIJTBRHEPOZOPLOA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.271476166 (568h40m25.142559211s) [noise=true , address=45.91.116.5:4298 , piece_id=IEK4NDC3QHPKKIZUTWQOQX6YOUOLDFXZIFKW4B6REL4YFHFERRWQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.271902749 (568h40m25.142985784s) [noise=true , address=145.79.227.6:4258 , piece_id=7KILQAL33HL5AR6D5CXTUJMLLYGRDUPTMDWEKSBJMVYHEUQ6PBXQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.272525764 (568h40m25.143608799s) [noise=true , address=45.91.116.12:4226 , piece_id=UPEQKT2AR7GUHUSPFSOHIAER7C25W7S7FBDYHPXOSPTGIOBFK57Q, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.273131338 (568h40m25.144214383s) [noise=true , address=145.79.227.15:4876 , piece_id=4CBROXUP33L2G2SWOTRR2H7BVUW3AWKLJTLTACMHJYHYTCL5WOHQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.274475868 (568h40m25.145558913s) [noise=true , address=45.91.116.4:4939 , piece_id=JW722VLZ72R6T6VAMFRQ7KCJNTD5FQ5B7DBATDSY5YLUEHCHIX3A, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:26.973964979 (568h40m26.845048024s) [upload=498563]: Next returned nil segment Dec 29 08:09:26.973978093 (568h40m26.845061138s) [upload=498563]: Waiting for error group managing segments... Dec 29 08:09:27.257189477 (568h40m27.128272512s) [noise=true , address=145.79.227.18:4006 , piece_id=LQJZVYYWL62UD5P52QYG2YD4VKUTSK6UJIRELUOGUJ4CIX4ZPU2Q, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.257942331 (568h40m27.129025376s) [noise=true , address=145.79.227.2:4115 , piece_id=WZMXOUIJIGKLX4GDKNXX3NUZ4PDGN2RNQMQYE6J5UHFLPONG6NFA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.261055839 (568h40m27.132138884s) [noise=true , address=145.79.227.6:4258 , piece_id=7KILQAL33HL5AR6D5CXTUJMLLYGRDUPTMDWEKSBJMVYHEUQ6PBXQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.273716727 (568h40m27.144799762s) [noise=true , address=145.79.227.42:4121 , piece_id=TZJ27VQFY3J53WAGT5G2WVGSTIEEJYTU3JJOF6WVVHATHSTLL7DA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.279942462 (568h40m27.151025507s) [noise=true , address=145.79.227.15:4876 , piece_id=4CBROXUP33L2G2SWOTRR2H7BVUW3AWKLJTLTACMHJYHYTCL5WOHQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.293588555 (568h40m27.1646716s) [noise=true , address=145.79.227.31:4798 , piece_id=I32JWMKL6HDTQAREE3IDJKRVQYMN7GAHAZIFWG3YRKIV4HN3BSFQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.459347824 (568h40m27.330430859s) [noise=true , address=77.90.31.10:4519 , piece_id=OJVPUGJLS4D5OY7W37IGLV4MBOTYLLWA5VJ6FIR37MJO6NXWXE6Q, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.544810226 (568h40m27.415893271s) [noise=true , address=77.90.31.16:4735 , piece_id=CGXZ4ZDVW4R2BQZTTGBSW7HPURF2JJRFGWL2VGULRJN2F65LTEPQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.576419104 (568h40m27.447502149s) [noise=true , address=45.91.116.9:4262 , piece_id=TVILUK5NEJOMZDTEJPPDHNSXO7TSNEIWWVSOZLV6544MM46YSMTA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.621846654 (568h40m27.492929699s) [noise=true , address=45.91.116.4:4939 , piece_id=JW722VLZ72R6T6VAMFRQ7KCJNTD5FQ5B7DBATDSY5YLUEHCHIX3A, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.668727752 (568h40m27.539810797s) [noise=true , address=45.91.116.8:4047 , piece_id=2LSTDDF7PU2QZUCP2NSHI54U2MVZRXCXOOKETMUDYMGKGITLPU6A, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.872253141 (568h40m27.743336177s) [noise=true , address=45.91.116.12:4226 , piece_id=UPEQKT2AR7GUHUSPFSOHIAER7C25W7S7FBDYHPXOSPTGIOBFK57Q, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:28.200848121 (568h40m28.071931156s) [noise=true , address=77.90.31.18:4805 , piece_id=7BN4ZP6QKPNXQS2RWYDM5GY773UBS3E73MY2IL2R55SDWYF4B4UQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:28.200887066 (568h40m28.071970111s) [seg_pos={0 0}, upload=498563]: Segment reached optimal threshold of 13 pieces. Dec 29 08:09:28.201151669 (568h40m28.072234714s) [noise=true , address=45.91.116.11:4120 , piece_id=LJQV4BLS2ASIVWJE5KEB4OQY3BDHH3DEQ6N2FIJTBRHEPOZOPLOA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 08:09:28.201160676 (568h40m28.072243721s) [noise=true , address=45.91.116.11:4120 , piece_id=LJQV4BLS2ASIVWJE5KEB4OQY3BDHH3DEQ6N2FIJTBRHEPOZOPLOA, seg_pos={0 0}, upload=498563]: Long tail cancellation due to optimal threshold Dec 29 08:09:28.201399410 (568h40m28.072482456s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRG42GDDRJER47YL5H6AOL5VSJ4GSK6CCY4PRRSOHVMZ7VHK3PQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 29 08:09:28.201421513 (568h40m28.072504568s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRG42GDDRJER47YL5H6AOL5VSJ4GSK6CCY4PRRSOHVMZ7VHK3PQ, seg_pos={0 0}, upload=498563]: Long tail cancellation due to optimal threshold Dec 29 08:09:28.201562211 (568h40m28.072645256s) [noise=true , address=45.91.116.5:4298 , piece_id=IEK4NDC3QHPKKIZUTWQOQX6YOUOLDFXZIFKW4B6REL4YFHFERRWQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 08:09:28.201575025 (568h40m28.07265807s) [noise=true , address=45.91.116.5:4298 , piece_id=IEK4NDC3QHPKKIZUTWQOQX6YOUOLDFXZIFKW4B6REL4YFHFERRWQ, seg_pos={0 0}, upload=498563]: Long tail cancellation due to optimal threshold Dec 29 08:09:28.201600333 (568h40m28.072683378s) [seg_pos={0 0}, upload=498563]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:09:28.201687941 (568h40m28.072770976s) [upload=498563]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:09:28.201694874 (568h40m28.072777909s) [upload=498563]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:09:28.496547263 (568h40m28.367630308s) [upload=498563]: Done uploading segments... Dec 29 08:09:28.496568594 (568h40m28.367651639s) [upload=498563]: Upload finished. err: <nil> Duration: 5.923726142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..............X.X.X.............X.......................] duration: 15.350230317s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:11:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 15.350249815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..............X.X.X.............X......................] duration: 47.436198685s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:11:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 47.436224684s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................X........................X..X......X.....] duration: 4.425025526s trace: https://jaeger.dev.storj.io/trace/43163810b1d7c614 FAIL (2025-12-29 08:11:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.425049441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X...X..XX..X...X.XXXX.X........X...X............XX....X.X.] duration: 5.917816791s trace: https://jaeger.dev.storj.io/trace/6a66a6c26a16039b FAIL (2025-12-29 08:10:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: TVILUK5NEJOMZDTEJPPDHNSXO7TSNEIWWVSOZLV6544MM46YSMTA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 7BN4ZP6QKPNXQS2RWYDM5GY773UBS3E73MY2IL2R55SDWYF4B4UQ) context deadline exceeded ======LOGS====== Dec 29 08:09:25.020604922 (568h40m24.891687957s) [upload=498563]: Starting upload Dec 29 08:09:25.020673453 (568h40m24.891756488s) [upload=498563]: Done starting upload Dec 29 08:09:25.020864347 (568h40m24.891947382s) [upload=498563]: Uploading segments... Dec 29 08:09:25.020894374 (568h40m24.891977409s) [upload=498563]: Got next segment. Inline: false Dec 29 08:09:25.020929842 (568h40m24.892012887s) [upload=498563]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:09:25.020935292 (568h40m24.892018327s) [upload=498563]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:09:25.266662981 (568h40m25.137746016s) [seg_pos={0 0}, upload=498563]: Begin upload segment... Dec 29 08:09:25.266701785 (568h40m25.13778482s) [seg_pos={0 0}, upload=498563]: Done begin upload segment. Dec 29 08:09:25.266718437 (568h40m25.137801472s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRG42GDDRJER47YL5H6AOL5VSJ4GSK6CCY4PRRSOHVMZ7VHK3PQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.266727664 (568h40m25.137810709s) [noise=true , address=145.79.227.18:4006 , piece_id=LQJZVYYWL62UD5P52QYG2YD4VKUTSK6UJIRELUOGUJ4CIX4ZPU2Q, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.267179214 (568h40m25.138262249s) [noise=true , address=45.91.116.8:4047 , piece_id=2LSTDDF7PU2QZUCP2NSHI54U2MVZRXCXOOKETMUDYMGKGITLPU6A, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.267840043 (568h40m25.138923078s) [noise=true , address=145.79.227.2:4115 , piece_id=WZMXOUIJIGKLX4GDKNXX3NUZ4PDGN2RNQMQYE6J5UHFLPONG6NFA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268114956 (568h40m25.139197991s) [noise=true , address=77.90.31.18:4805 , piece_id=7BN4ZP6QKPNXQS2RWYDM5GY773UBS3E73MY2IL2R55SDWYF4B4UQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268400049 (568h40m25.139483084s) [noise=true , address=145.79.227.31:4798 , piece_id=I32JWMKL6HDTQAREE3IDJKRVQYMN7GAHAZIFWG3YRKIV4HN3BSFQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268667088 (568h40m25.139750133s) [noise=true , address=77.90.31.16:4735 , piece_id=CGXZ4ZDVW4R2BQZTTGBSW7HPURF2JJRFGWL2VGULRJN2F65LTEPQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268930779 (568h40m25.140013814s) [noise=true , address=145.79.227.42:4121 , piece_id=TZJ27VQFY3J53WAGT5G2WVGSTIEEJYTU3JJOF6WVVHATHSTLL7DA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.269238936 (568h40m25.140321971s) [noise=true , address=45.91.116.9:4262 , piece_id=TVILUK5NEJOMZDTEJPPDHNSXO7TSNEIWWVSOZLV6544MM46YSMTA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.270262556 (568h40m25.141345601s) [noise=true , address=77.90.31.10:4519 , piece_id=OJVPUGJLS4D5OY7W37IGLV4MBOTYLLWA5VJ6FIR37MJO6NXWXE6Q, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.270590490 (568h40m25.141673525s) [noise=true , address=45.91.116.11:4120 , piece_id=LJQV4BLS2ASIVWJE5KEB4OQY3BDHH3DEQ6N2FIJTBRHEPOZOPLOA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.271476166 (568h40m25.142559211s) [noise=true , address=45.91.116.5:4298 , piece_id=IEK4NDC3QHPKKIZUTWQOQX6YOUOLDFXZIFKW4B6REL4YFHFERRWQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.271902749 (568h40m25.142985784s) [noise=true , address=145.79.227.6:4258 , piece_id=7KILQAL33HL5AR6D5CXTUJMLLYGRDUPTMDWEKSBJMVYHEUQ6PBXQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.272525764 (568h40m25.143608799s) [noise=true , address=45.91.116.12:4226 , piece_id=UPEQKT2AR7GUHUSPFSOHIAER7C25W7S7FBDYHPXOSPTGIOBFK57Q, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.273131338 (568h40m25.144214383s) [noise=true , address=145.79.227.15:4876 , piece_id=4CBROXUP33L2G2SWOTRR2H7BVUW3AWKLJTLTACMHJYHYTCL5WOHQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.274475868 (568h40m25.145558913s) [noise=true , address=45.91.116.4:4939 , piece_id=JW722VLZ72R6T6VAMFRQ7KCJNTD5FQ5B7DBATDSY5YLUEHCHIX3A, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:26.973964979 (568h40m26.845048024s) [upload=498563]: Next returned nil segment Dec 29 08:09:26.973978093 (568h40m26.845061138s) [upload=498563]: Waiting for error group managing segments... Dec 29 08:09:27.257189477 (568h40m27.128272512s) [noise=true , address=145.79.227.18:4006 , piece_id=LQJZVYYWL62UD5P52QYG2YD4VKUTSK6UJIRELUOGUJ4CIX4ZPU2Q, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.257942331 (568h40m27.129025376s) [noise=true , address=145.79.227.2:4115 , piece_id=WZMXOUIJIGKLX4GDKNXX3NUZ4PDGN2RNQMQYE6J5UHFLPONG6NFA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.261055839 (568h40m27.132138884s) [noise=true , address=145.79.227.6:4258 , piece_id=7KILQAL33HL5AR6D5CXTUJMLLYGRDUPTMDWEKSBJMVYHEUQ6PBXQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.273716727 (568h40m27.144799762s) [noise=true , address=145.79.227.42:4121 , piece_id=TZJ27VQFY3J53WAGT5G2WVGSTIEEJYTU3JJOF6WVVHATHSTLL7DA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.279942462 (568h40m27.151025507s) [noise=true , address=145.79.227.15:4876 , piece_id=4CBROXUP33L2G2SWOTRR2H7BVUW3AWKLJTLTACMHJYHYTCL5WOHQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.293588555 (568h40m27.1646716s) [noise=true , address=145.79.227.31:4798 , piece_id=I32JWMKL6HDTQAREE3IDJKRVQYMN7GAHAZIFWG3YRKIV4HN3BSFQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.459347824 (568h40m27.330430859s) [noise=true , address=77.90.31.10:4519 , piece_id=OJVPUGJLS4D5OY7W37IGLV4MBOTYLLWA5VJ6FIR37MJO6NXWXE6Q, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.544810226 (568h40m27.415893271s) [noise=true , address=77.90.31.16:4735 , piece_id=CGXZ4ZDVW4R2BQZTTGBSW7HPURF2JJRFGWL2VGULRJN2F65LTEPQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.576419104 (568h40m27.447502149s) [noise=true , address=45.91.116.9:4262 , piece_id=TVILUK5NEJOMZDTEJPPDHNSXO7TSNEIWWVSOZLV6544MM46YSMTA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.621846654 (568h40m27.492929699s) [noise=true , address=45.91.116.4:4939 , piece_id=JW722VLZ72R6T6VAMFRQ7KCJNTD5FQ5B7DBATDSY5YLUEHCHIX3A, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.668727752 (568h40m27.539810797s) [noise=true , address=45.91.116.8:4047 , piece_id=2LSTDDF7PU2QZUCP2NSHI54U2MVZRXCXOOKETMUDYMGKGITLPU6A, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.872253141 (568h40m27.743336177s) [noise=true , address=45.91.116.12:4226 , piece_id=UPEQKT2AR7GUHUSPFSOHIAER7C25W7S7FBDYHPXOSPTGIOBFK57Q, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:28.200848121 (568h40m28.071931156s) [noise=true , address=77.90.31.18:4805 , piece_id=7BN4ZP6QKPNXQS2RWYDM5GY773UBS3E73MY2IL2R55SDWYF4B4UQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:28.200887066 (568h40m28.071970111s) [seg_pos={0 0}, upload=498563]: Segment reached optimal threshold of 13 pieces. Dec 29 08:09:28.201151669 (568h40m28.072234714s) [noise=true , address=45.91.116.11:4120 , piece_id=LJQV4BLS2ASIVWJE5KEB4OQY3BDHH3DEQ6N2FIJTBRHEPOZOPLOA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 08:09:28.201160676 (568h40m28.072243721s) [noise=true , address=45.91.116.11:4120 , piece_id=LJQV4BLS2ASIVWJE5KEB4OQY3BDHH3DEQ6N2FIJTBRHEPOZOPLOA, seg_pos={0 0}, upload=498563]: Long tail cancellation due to optimal threshold Dec 29 08:09:28.201399410 (568h40m28.072482456s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRG42GDDRJER47YL5H6AOL5VSJ4GSK6CCY4PRRSOHVMZ7VHK3PQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 29 08:09:28.201421513 (568h40m28.072504568s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRG42GDDRJER47YL5H6AOL5VSJ4GSK6CCY4PRRSOHVMZ7VHK3PQ, seg_pos={0 0}, upload=498563]: Long tail cancellation due to optimal threshold Dec 29 08:09:28.201562211 (568h40m28.072645256s) [noise=true , address=45.91.116.5:4298 , piece_id=IEK4NDC3QHPKKIZUTWQOQX6YOUOLDFXZIFKW4B6REL4YFHFERRWQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 08:09:28.201575025 (568h40m28.07265807s) [noise=true , address=45.91.116.5:4298 , piece_id=IEK4NDC3QHPKKIZUTWQOQX6YOUOLDFXZIFKW4B6REL4YFHFERRWQ, seg_pos={0 0}, upload=498563]: Long tail cancellation due to optimal threshold Dec 29 08:09:28.201600333 (568h40m28.072683378s) [seg_pos={0 0}, upload=498563]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:09:28.201687941 (568h40m28.072770976s) [upload=498563]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:09:28.201694874 (568h40m28.072777909s) [upload=498563]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:09:28.496547263 (568h40m28.367630308s) [upload=498563]: Done uploading segments... Dec 29 08:09:28.496568594 (568h40m28.367651639s) [upload=498563]: Upload finished. err: <nil> Duration: 5.917832561s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................X........................X..X......X....] duration: 3.364392648s trace: https://jaeger.dev.storj.io/trace/43163810b1d7c614 FAIL (2025-12-29 08:10:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.364555366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..............X.X.X.............X.....................] duration: 54.484321154s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:10:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 54.484345861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X...X..XX..X...X.XXXX.X........X...X............XX....X.X] duration: 30.004585738s trace: https://jaeger.dev.storj.io/trace/6a66a6c26a16039b FAIL (2025-12-29 08:09:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: TVILUK5NEJOMZDTEJPPDHNSXO7TSNEIWWVSOZLV6544MM46YSMTA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 7BN4ZP6QKPNXQS2RWYDM5GY773UBS3E73MY2IL2R55SDWYF4B4UQ) context deadline exceeded ======LOGS====== Dec 29 08:09:25.020604922 (568h40m24.891687957s) [upload=498563]: Starting upload Dec 29 08:09:25.020673453 (568h40m24.891756488s) [upload=498563]: Done starting upload Dec 29 08:09:25.020864347 (568h40m24.891947382s) [upload=498563]: Uploading segments... Dec 29 08:09:25.020894374 (568h40m24.891977409s) [upload=498563]: Got next segment. Inline: false Dec 29 08:09:25.020929842 (568h40m24.892012887s) [upload=498563]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:09:25.020935292 (568h40m24.892018327s) [upload=498563]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:09:25.266662981 (568h40m25.137746016s) [seg_pos={0 0}, upload=498563]: Begin upload segment... Dec 29 08:09:25.266701785 (568h40m25.13778482s) [seg_pos={0 0}, upload=498563]: Done begin upload segment. Dec 29 08:09:25.266718437 (568h40m25.137801472s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRG42GDDRJER47YL5H6AOL5VSJ4GSK6CCY4PRRSOHVMZ7VHK3PQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.266727664 (568h40m25.137810709s) [noise=true , address=145.79.227.18:4006 , piece_id=LQJZVYYWL62UD5P52QYG2YD4VKUTSK6UJIRELUOGUJ4CIX4ZPU2Q, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.267179214 (568h40m25.138262249s) [noise=true , address=45.91.116.8:4047 , piece_id=2LSTDDF7PU2QZUCP2NSHI54U2MVZRXCXOOKETMUDYMGKGITLPU6A, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.267840043 (568h40m25.138923078s) [noise=true , address=145.79.227.2:4115 , piece_id=WZMXOUIJIGKLX4GDKNXX3NUZ4PDGN2RNQMQYE6J5UHFLPONG6NFA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268114956 (568h40m25.139197991s) [noise=true , address=77.90.31.18:4805 , piece_id=7BN4ZP6QKPNXQS2RWYDM5GY773UBS3E73MY2IL2R55SDWYF4B4UQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268400049 (568h40m25.139483084s) [noise=true , address=145.79.227.31:4798 , piece_id=I32JWMKL6HDTQAREE3IDJKRVQYMN7GAHAZIFWG3YRKIV4HN3BSFQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268667088 (568h40m25.139750133s) [noise=true , address=77.90.31.16:4735 , piece_id=CGXZ4ZDVW4R2BQZTTGBSW7HPURF2JJRFGWL2VGULRJN2F65LTEPQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.268930779 (568h40m25.140013814s) [noise=true , address=145.79.227.42:4121 , piece_id=TZJ27VQFY3J53WAGT5G2WVGSTIEEJYTU3JJOF6WVVHATHSTLL7DA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.269238936 (568h40m25.140321971s) [noise=true , address=45.91.116.9:4262 , piece_id=TVILUK5NEJOMZDTEJPPDHNSXO7TSNEIWWVSOZLV6544MM46YSMTA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.270262556 (568h40m25.141345601s) [noise=true , address=77.90.31.10:4519 , piece_id=OJVPUGJLS4D5OY7W37IGLV4MBOTYLLWA5VJ6FIR37MJO6NXWXE6Q, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.270590490 (568h40m25.141673525s) [noise=true , address=45.91.116.11:4120 , piece_id=LJQV4BLS2ASIVWJE5KEB4OQY3BDHH3DEQ6N2FIJTBRHEPOZOPLOA, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.271476166 (568h40m25.142559211s) [noise=true , address=45.91.116.5:4298 , piece_id=IEK4NDC3QHPKKIZUTWQOQX6YOUOLDFXZIFKW4B6REL4YFHFERRWQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.271902749 (568h40m25.142985784s) [noise=true , address=145.79.227.6:4258 , piece_id=7KILQAL33HL5AR6D5CXTUJMLLYGRDUPTMDWEKSBJMVYHEUQ6PBXQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.272525764 (568h40m25.143608799s) [noise=true , address=45.91.116.12:4226 , piece_id=UPEQKT2AR7GUHUSPFSOHIAER7C25W7S7FBDYHPXOSPTGIOBFK57Q, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.273131338 (568h40m25.144214383s) [noise=true , address=145.79.227.15:4876 , piece_id=4CBROXUP33L2G2SWOTRR2H7BVUW3AWKLJTLTACMHJYHYTCL5WOHQ, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:25.274475868 (568h40m25.145558913s) [noise=true , address=45.91.116.4:4939 , piece_id=JW722VLZ72R6T6VAMFRQ7KCJNTD5FQ5B7DBATDSY5YLUEHCHIX3A, seg_pos={0 0}, upload=498563]: Uploading piece... Dec 29 08:09:26.973964979 (568h40m26.845048024s) [upload=498563]: Next returned nil segment Dec 29 08:09:26.973978093 (568h40m26.845061138s) [upload=498563]: Waiting for error group managing segments... Dec 29 08:09:27.257189477 (568h40m27.128272512s) [noise=true , address=145.79.227.18:4006 , piece_id=LQJZVYYWL62UD5P52QYG2YD4VKUTSK6UJIRELUOGUJ4CIX4ZPU2Q, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.257942331 (568h40m27.129025376s) [noise=true , address=145.79.227.2:4115 , piece_id=WZMXOUIJIGKLX4GDKNXX3NUZ4PDGN2RNQMQYE6J5UHFLPONG6NFA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.261055839 (568h40m27.132138884s) [noise=true , address=145.79.227.6:4258 , piece_id=7KILQAL33HL5AR6D5CXTUJMLLYGRDUPTMDWEKSBJMVYHEUQ6PBXQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.273716727 (568h40m27.144799762s) [noise=true , address=145.79.227.42:4121 , piece_id=TZJ27VQFY3J53WAGT5G2WVGSTIEEJYTU3JJOF6WVVHATHSTLL7DA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.279942462 (568h40m27.151025507s) [noise=true , address=145.79.227.15:4876 , piece_id=4CBROXUP33L2G2SWOTRR2H7BVUW3AWKLJTLTACMHJYHYTCL5WOHQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.293588555 (568h40m27.1646716s) [noise=true , address=145.79.227.31:4798 , piece_id=I32JWMKL6HDTQAREE3IDJKRVQYMN7GAHAZIFWG3YRKIV4HN3BSFQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.459347824 (568h40m27.330430859s) [noise=true , address=77.90.31.10:4519 , piece_id=OJVPUGJLS4D5OY7W37IGLV4MBOTYLLWA5VJ6FIR37MJO6NXWXE6Q, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.544810226 (568h40m27.415893271s) [noise=true , address=77.90.31.16:4735 , piece_id=CGXZ4ZDVW4R2BQZTTGBSW7HPURF2JJRFGWL2VGULRJN2F65LTEPQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.576419104 (568h40m27.447502149s) [noise=true , address=45.91.116.9:4262 , piece_id=TVILUK5NEJOMZDTEJPPDHNSXO7TSNEIWWVSOZLV6544MM46YSMTA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.621846654 (568h40m27.492929699s) [noise=true , address=45.91.116.4:4939 , piece_id=JW722VLZ72R6T6VAMFRQ7KCJNTD5FQ5B7DBATDSY5YLUEHCHIX3A, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.668727752 (568h40m27.539810797s) [noise=true , address=45.91.116.8:4047 , piece_id=2LSTDDF7PU2QZUCP2NSHI54U2MVZRXCXOOKETMUDYMGKGITLPU6A, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:27.872253141 (568h40m27.743336177s) [noise=true , address=45.91.116.12:4226 , piece_id=UPEQKT2AR7GUHUSPFSOHIAER7C25W7S7FBDYHPXOSPTGIOBFK57Q, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:28.200848121 (568h40m28.071931156s) [noise=true , address=77.90.31.18:4805 , piece_id=7BN4ZP6QKPNXQS2RWYDM5GY773UBS3E73MY2IL2R55SDWYF4B4UQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: <nil> Dec 29 08:09:28.200887066 (568h40m28.071970111s) [seg_pos={0 0}, upload=498563]: Segment reached optimal threshold of 13 pieces. Dec 29 08:09:28.201151669 (568h40m28.072234714s) [noise=true , address=45.91.116.11:4120 , piece_id=LJQV4BLS2ASIVWJE5KEB4OQY3BDHH3DEQ6N2FIJTBRHEPOZOPLOA, seg_pos={0 0}, upload=498563]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 08:09:28.201160676 (568h40m28.072243721s) [noise=true , address=45.91.116.11:4120 , piece_id=LJQV4BLS2ASIVWJE5KEB4OQY3BDHH3DEQ6N2FIJTBRHEPOZOPLOA, seg_pos={0 0}, upload=498563]: Long tail cancellation due to optimal threshold Dec 29 08:09:28.201399410 (568h40m28.072482456s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRG42GDDRJER47YL5H6AOL5VSJ4GSK6CCY4PRRSOHVMZ7VHK3PQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 29 08:09:28.201421513 (568h40m28.072504568s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRG42GDDRJER47YL5H6AOL5VSJ4GSK6CCY4PRRSOHVMZ7VHK3PQ, seg_pos={0 0}, upload=498563]: Long tail cancellation due to optimal threshold Dec 29 08:09:28.201562211 (568h40m28.072645256s) [noise=true , address=45.91.116.5:4298 , piece_id=IEK4NDC3QHPKKIZUTWQOQX6YOUOLDFXZIFKW4B6REL4YFHFERRWQ, seg_pos={0 0}, upload=498563]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 08:09:28.201575025 (568h40m28.07265807s) [noise=true , address=45.91.116.5:4298 , piece_id=IEK4NDC3QHPKKIZUTWQOQX6YOUOLDFXZIFKW4B6REL4YFHFERRWQ, seg_pos={0 0}, upload=498563]: Long tail cancellation due to optimal threshold Dec 29 08:09:28.201600333 (568h40m28.072683378s) [seg_pos={0 0}, upload=498563]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:09:28.201687941 (568h40m28.072770976s) [upload=498563]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:09:28.201694874 (568h40m28.072777909s) [upload=498563]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:09:28.496547263 (568h40m28.367630308s) [upload=498563]: Done uploading segments... Dec 29 08:09:28.496568594 (568h40m28.367651639s) [upload=498563]: Upload finished. err: <nil> Duration: 30.004607831s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................X........................X..X......X...] duration: 3.463530036s trace: https://jaeger.dev.storj.io/trace/43163810b1d7c614 FAIL (2025-12-29 08:09:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463556807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..............X.X.X.............X....................] duration: 28.960335656s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:09:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 28.960349623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..............X.X.X.............X...................] duration: 49.015082807s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:08:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 49.015117864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................X........................X..X......X..] duration: 4.138275551s trace: https://jaeger.dev.storj.io/trace/43163810b1d7c614 FAIL (2025-12-29 08:08:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.138293505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX..X...X..XX..X...X.XXXX.X........X...X............XX....X.] duration: 5.933440558s trace: https://jaeger.dev.storj.io/trace/2c99ba530e533ee3 FAIL (2025-12-29 08:08:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 6JYY6QKYDSRUDCZOT7NGTDMW2BGK3JHGVUUO34VQAZRSOOBYNXMA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 3JGXN7SVIC4G3NQQOZL6NFNQ2V6XK2KINALDCUAYUG433NWZNUEQ) context deadline exceeded ======LOGS====== 29 08:07:44.476610954 (568h38m44.347693989s) [upload=498543]: Starting upload Dec 29 08:07:44.476700375 (568h38m44.34778341s) [upload=498543]: Done starting upload Dec 29 08:07:44.476928599 (568h38m44.348011634s) [upload=498543]: Uploading segments... Dec 29 08:07:44.477006528 (568h38m44.348089563s) [upload=498543]: Got next segment. Inline: false Dec 29 08:07:44.477041124 (568h38m44.348124169s) [upload=498543]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:07:44.477059969 (568h38m44.348143014s) [upload=498543]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:07:44.724164670 (568h38m44.595247726s) [seg_pos={0 0}, upload=498543]: Begin upload segment... Dec 29 08:07:44.724270011 (568h38m44.595353056s) [seg_pos={0 0}, upload=498543]: Done begin upload segment. Dec 29 08:07:44.724334304 (568h38m44.595417349s) [noise=true , address=77.90.31.15:4700 , piece_id=4WGLCCL6SG5ZFGXPJMSOYMZVKFJUMXZGPRD75VI5DKNSLXVYVA2A, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.724339884 (568h38m44.595422919s) [noise=true , address=45.91.116.14:4189 , piece_id=6JYY6QKYDSRUDCZOT7NGTDMW2BGK3JHGVUUO34VQAZRSOOBYNXMA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.725158143 (568h38m44.596241178s) [noise=true , address=145.79.227.36:4517 , piece_id=I2XQTHHNCLL7JE4G3YZK57MLDAGKZFDRBXWJJ42MXBKFPVUEC24Q, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.725543817 (568h38m44.596626852s) [noise=true , address=77.90.31.11:4555 , piece_id=W2B77TCGGJBPNQ5BPU3R23MBFQVGQ3JP3WIGTDPROLCLSTTCXDQQ, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.725854649 (568h38m44.596937694s) [noise=true , address=45.91.116.5:4298 , piece_id=KUE72SYO3YVC5QYUULTNNOUDX22UQXADXZQNL2GDOCJNHA4MZ7KA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.726145673 (568h38m44.597228718s) [noise=true , address=145.79.227.5:4222 , piece_id=DNGIMZNDGXJCKXQNKOZJV7LQ7TER7YSOYHOCIKPCT3WHE2EBAZBQ, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.726247607 (568h38m44.597330652s) [noise=true , address=145.79.227.3:4150 , piece_id=3XUW37XACEVKDUD32MFAQKWCEH6N4FBLXY234XHUB2TD66YLHYSA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.726500608 (568h38m44.597583643s) [noise=true , address=145.79.227.48:4302 , piece_id=KNLQ4TSH6PYV7TGRS73NGSDYCN3ZS6CNYY7QGCRYOX5EPLKS4ZYA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.726817001 (568h38m44.597900036s) [noise=true , address=77.90.31.21:4880 , piece_id=3JGXN7SVIC4G3NQQOZL6NFNQ2V6XK2KINALDCUAYUG433NWZNUEQ, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.727170694 (568h38m44.598253739s) [noise=true , address=145.79.227.23:4547 , piece_id=ZWM6DGZ2A4T7CSGF2FQZMH7MFENJG22GIMG5MORDLB4PILRR6LYA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.727684984 (568h38m44.598768019s) [noise=true , address=145.79.227.42:4123 , piece_id=BWRNRLPBPHPU3UXQL4XVTMCQ26JFO2RPOKOO3VQUY7KUUDI4GY4A, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.728153706 (568h38m44.599236741s) [noise=true , address=145.79.227.26:4655 , piece_id=6BCNUXQFTUU56THSZKCPTIEJ6OMENN4UUXOGG75GVAMSG2CTL3TA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.728599865 (568h38m44.599682911s) [noise=true , address=145.79.227.46:4601 , piece_id=EX7EAFHXGNUVYHF3DJZBX72N4NOFE4ERMK42VWEFGTJQUG7BE4OA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.729096501 (568h38m44.600179546s) [noise=true , address=145.79.227.14:4840 , piece_id=BX4IAN3OZKIVRXV4DDSVKH6TG74DWCYL7QEIZEFIFCT4L2R6EPRA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.729592025 (568h38m44.60067506s) [noise=true , address=145.79.227.25:4617 , piece_id=ZSY3ULAHU7TBYUO4JJ7MI24CJYR3ANLAKOAXKOHYIPDQOX3S4WCA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.731132258 (568h38m44.602215293s) [noise=true , address=145.79.227.37:4567 , piece_id=KRRUVPU2OWWNHQLHFIB2IOTTB3UCXUVCGPB4JWBLBOPLAW23W2WQ, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:46.437092482 (568h38m46.308175517s) [upload=498543]: Next returned nil segment Dec 29 08:07:46.437106899 (568h38m46.308189944s) [upload=498543]: Waiting for error group managing segments... Dec 29 08:07:46.718049102 (568h38m46.589132137s) [noise=true , address=145.79.227.3:4150 , piece_id=3XUW37XACEVKDUD32MFAQKWCEH6N4FBLXY234XHUB2TD66YLHYSA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.725380162 (568h38m46.596463197s) [noise=true , address=145.79.227.42:4123 , piece_id=BWRNRLPBPHPU3UXQL4XVTMCQ26JFO2RPOKOO3VQUY7KUUDI4GY4A, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.726780318 (568h38m46.597863353s) [noise=true , address=145.79.227.46:4601 , piece_id=EX7EAFHXGNUVYHF3DJZBX72N4NOFE4ERMK42VWEFGTJQUG7BE4OA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.728362620 (568h38m46.599445665s) [noise=true , address=145.79.227.26:4655 , piece_id=6BCNUXQFTUU56THSZKCPTIEJ6OMENN4UUXOGG75GVAMSG2CTL3TA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.732249712 (568h38m46.603332747s) [noise=true , address=145.79.227.23:4547 , piece_id=ZWM6DGZ2A4T7CSGF2FQZMH7MFENJG22GIMG5MORDLB4PILRR6LYA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.734338589 (568h38m46.605421634s) [noise=true , address=145.79.227.48:4302 , piece_id=KNLQ4TSH6PYV7TGRS73NGSDYCN3ZS6CNYY7QGCRYOX5EPLKS4ZYA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.734726809 (568h38m46.605809844s) [noise=true , address=145.79.227.25:4617 , piece_id=ZSY3ULAHU7TBYUO4JJ7MI24CJYR3ANLAKOAXKOHYIPDQOX3S4WCA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.739724024 (568h38m46.610807069s) [noise=true , address=145.79.227.37:4567 , piece_id=KRRUVPU2OWWNHQLHFIB2IOTTB3UCXUVCGPB4JWBLBOPLAW23W2WQ, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.741605607 (568h38m46.612688652s) [noise=true , address=145.79.227.36:4517 , piece_id=I2XQTHHNCLL7JE4G3YZK57MLDAGKZFDRBXWJJ42MXBKFPVUEC24Q, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.743168583 (568h38m46.614251618s) [noise=true , address=145.79.227.14:4840 , piece_id=BX4IAN3OZKIVRXV4DDSVKH6TG74DWCYL7QEIZEFIFCT4L2R6EPRA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.757232152 (568h38m46.628315197s) [noise=true , address=145.79.227.5:4222 , piece_id=DNGIMZNDGXJCKXQNKOZJV7LQ7TER7YSOYHOCIKPCT3WHE2EBAZBQ, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.962745618 (568h38m46.833828663s) [noise=true , address=77.90.31.21:4880 , piece_id=3JGXN7SVIC4G3NQQOZL6NFNQ2V6XK2KINALDCUAYUG433NWZNUEQ, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:47.038069315 (568h38m46.90915236s) [noise=true , address=45.91.116.14:4189 , piece_id=6JYY6QKYDSRUDCZOT7NGTDMW2BGK3JHGVUUO34VQAZRSOOBYNXMA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:47.038088972 (568h38m46.909172007s) [seg_pos={0 0}, upload=498543]: Segment reached optimal threshold of 13 pieces. Dec 29 08:07:47.038321215 (568h38m46.90940429s) [noise=true , address=45.91.116.5:4298 , piece_id=KUE72SYO3YVC5QYUULTNNOUDX22UQXADXZQNL2GDOCJNHA4MZ7KA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 08:07:47.038334320 (568h38m46.909417685s) [noise=true , address=45.91.116.5:4298 , piece_id=KUE72SYO3YVC5QYUULTNNOUDX22UQXADXZQNL2GDOCJNHA4MZ7KA, seg_pos={0 0}, upload=498543]: Long tail cancellation due to optimal threshold Dec 29 08:07:47.038446864 (568h38m46.909529899s) [noise=true , address=77.90.31.15:4700 , piece_id=4WGLCCL6SG5ZFGXPJMSOYMZVKFJUMXZGPRD75VI5DKNSLXVYVA2A, seg_pos={0 0}, upload=498543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:55792->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:55792->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55792->77.90.31.15:4700: use of closed network connection Dec 29 08:07:47.038478614 (568h38m46.909561669s) [noise=true , address=77.90.31.15:4700 , piece_id=4WGLCCL6SG5ZFGXPJMSOYMZVKFJUMXZGPRD75VI5DKNSLXVYVA2A, seg_pos={0 0}, upload=498543]: Long tail cancellation due to optimal threshold Dec 29 08:07:47.038820115 (568h38m46.90990317s) [noise=true , address=77.90.31.11:4555 , piece_id=W2B77TCGGJBPNQ5BPU3R23MBFQVGQ3JP3WIGTDPROLCLSTTCXDQQ, seg_pos={0 0}, upload=498543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 29 08:07:47.038831827 (568h38m46.909914872s) [noise=true , address=77.90.31.11:4555 , piece_id=W2B77TCGGJBPNQ5BPU3R23MBFQVGQ3JP3WIGTDPROLCLSTTCXDQQ, seg_pos={0 0}, upload=498543]: Long tail cancellation due to optimal threshold Dec 29 08:07:47.038860321 (568h38m46.909943366s) [seg_pos={0 0}, upload=498543]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:07:47.038954901 (568h38m46.910037936s) [upload=498543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:07:47.038960833 (568h38m46.910043878s) [upload=498543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:07:47.329690392 (568h38m47.200773447s) [upload=498543]: Done uploading segments... Dec 29 08:07:47.329714879 (568h38m47.200797924s) [upload=498543]: Upload finished. err: <nil> Duration: 5.933456319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX..X...X..XX..X...X.XXXX.X........X...X............XX....X] duration: 30.002292249s trace: https://jaeger.dev.storj.io/trace/2c99ba530e533ee3 FAIL (2025-12-29 08:08:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 6JYY6QKYDSRUDCZOT7NGTDMW2BGK3JHGVUUO34VQAZRSOOBYNXMA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 3JGXN7SVIC4G3NQQOZL6NFNQ2V6XK2KINALDCUAYUG433NWZNUEQ) context deadline exceeded ======LOGS====== 29 08:07:44.476610954 (568h38m44.347693989s) [upload=498543]: Starting upload Dec 29 08:07:44.476700375 (568h38m44.34778341s) [upload=498543]: Done starting upload Dec 29 08:07:44.476928599 (568h38m44.348011634s) [upload=498543]: Uploading segments... Dec 29 08:07:44.477006528 (568h38m44.348089563s) [upload=498543]: Got next segment. Inline: false Dec 29 08:07:44.477041124 (568h38m44.348124169s) [upload=498543]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:07:44.477059969 (568h38m44.348143014s) [upload=498543]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:07:44.724164670 (568h38m44.595247726s) [seg_pos={0 0}, upload=498543]: Begin upload segment... Dec 29 08:07:44.724270011 (568h38m44.595353056s) [seg_pos={0 0}, upload=498543]: Done begin upload segment. Dec 29 08:07:44.724334304 (568h38m44.595417349s) [noise=true , address=77.90.31.15:4700 , piece_id=4WGLCCL6SG5ZFGXPJMSOYMZVKFJUMXZGPRD75VI5DKNSLXVYVA2A, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.724339884 (568h38m44.595422919s) [noise=true , address=45.91.116.14:4189 , piece_id=6JYY6QKYDSRUDCZOT7NGTDMW2BGK3JHGVUUO34VQAZRSOOBYNXMA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.725158143 (568h38m44.596241178s) [noise=true , address=145.79.227.36:4517 , piece_id=I2XQTHHNCLL7JE4G3YZK57MLDAGKZFDRBXWJJ42MXBKFPVUEC24Q, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.725543817 (568h38m44.596626852s) [noise=true , address=77.90.31.11:4555 , piece_id=W2B77TCGGJBPNQ5BPU3R23MBFQVGQ3JP3WIGTDPROLCLSTTCXDQQ, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.725854649 (568h38m44.596937694s) [noise=true , address=45.91.116.5:4298 , piece_id=KUE72SYO3YVC5QYUULTNNOUDX22UQXADXZQNL2GDOCJNHA4MZ7KA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.726145673 (568h38m44.597228718s) [noise=true , address=145.79.227.5:4222 , piece_id=DNGIMZNDGXJCKXQNKOZJV7LQ7TER7YSOYHOCIKPCT3WHE2EBAZBQ, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.726247607 (568h38m44.597330652s) [noise=true , address=145.79.227.3:4150 , piece_id=3XUW37XACEVKDUD32MFAQKWCEH6N4FBLXY234XHUB2TD66YLHYSA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.726500608 (568h38m44.597583643s) [noise=true , address=145.79.227.48:4302 , piece_id=KNLQ4TSH6PYV7TGRS73NGSDYCN3ZS6CNYY7QGCRYOX5EPLKS4ZYA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.726817001 (568h38m44.597900036s) [noise=true , address=77.90.31.21:4880 , piece_id=3JGXN7SVIC4G3NQQOZL6NFNQ2V6XK2KINALDCUAYUG433NWZNUEQ, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.727170694 (568h38m44.598253739s) [noise=true , address=145.79.227.23:4547 , piece_id=ZWM6DGZ2A4T7CSGF2FQZMH7MFENJG22GIMG5MORDLB4PILRR6LYA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.727684984 (568h38m44.598768019s) [noise=true , address=145.79.227.42:4123 , piece_id=BWRNRLPBPHPU3UXQL4XVTMCQ26JFO2RPOKOO3VQUY7KUUDI4GY4A, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.728153706 (568h38m44.599236741s) [noise=true , address=145.79.227.26:4655 , piece_id=6BCNUXQFTUU56THSZKCPTIEJ6OMENN4UUXOGG75GVAMSG2CTL3TA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.728599865 (568h38m44.599682911s) [noise=true , address=145.79.227.46:4601 , piece_id=EX7EAFHXGNUVYHF3DJZBX72N4NOFE4ERMK42VWEFGTJQUG7BE4OA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.729096501 (568h38m44.600179546s) [noise=true , address=145.79.227.14:4840 , piece_id=BX4IAN3OZKIVRXV4DDSVKH6TG74DWCYL7QEIZEFIFCT4L2R6EPRA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.729592025 (568h38m44.60067506s) [noise=true , address=145.79.227.25:4617 , piece_id=ZSY3ULAHU7TBYUO4JJ7MI24CJYR3ANLAKOAXKOHYIPDQOX3S4WCA, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:44.731132258 (568h38m44.602215293s) [noise=true , address=145.79.227.37:4567 , piece_id=KRRUVPU2OWWNHQLHFIB2IOTTB3UCXUVCGPB4JWBLBOPLAW23W2WQ, seg_pos={0 0}, upload=498543]: Uploading piece... Dec 29 08:07:46.437092482 (568h38m46.308175517s) [upload=498543]: Next returned nil segment Dec 29 08:07:46.437106899 (568h38m46.308189944s) [upload=498543]: Waiting for error group managing segments... Dec 29 08:07:46.718049102 (568h38m46.589132137s) [noise=true , address=145.79.227.3:4150 , piece_id=3XUW37XACEVKDUD32MFAQKWCEH6N4FBLXY234XHUB2TD66YLHYSA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.725380162 (568h38m46.596463197s) [noise=true , address=145.79.227.42:4123 , piece_id=BWRNRLPBPHPU3UXQL4XVTMCQ26JFO2RPOKOO3VQUY7KUUDI4GY4A, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.726780318 (568h38m46.597863353s) [noise=true , address=145.79.227.46:4601 , piece_id=EX7EAFHXGNUVYHF3DJZBX72N4NOFE4ERMK42VWEFGTJQUG7BE4OA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.728362620 (568h38m46.599445665s) [noise=true , address=145.79.227.26:4655 , piece_id=6BCNUXQFTUU56THSZKCPTIEJ6OMENN4UUXOGG75GVAMSG2CTL3TA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.732249712 (568h38m46.603332747s) [noise=true , address=145.79.227.23:4547 , piece_id=ZWM6DGZ2A4T7CSGF2FQZMH7MFENJG22GIMG5MORDLB4PILRR6LYA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.734338589 (568h38m46.605421634s) [noise=true , address=145.79.227.48:4302 , piece_id=KNLQ4TSH6PYV7TGRS73NGSDYCN3ZS6CNYY7QGCRYOX5EPLKS4ZYA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.734726809 (568h38m46.605809844s) [noise=true , address=145.79.227.25:4617 , piece_id=ZSY3ULAHU7TBYUO4JJ7MI24CJYR3ANLAKOAXKOHYIPDQOX3S4WCA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.739724024 (568h38m46.610807069s) [noise=true , address=145.79.227.37:4567 , piece_id=KRRUVPU2OWWNHQLHFIB2IOTTB3UCXUVCGPB4JWBLBOPLAW23W2WQ, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.741605607 (568h38m46.612688652s) [noise=true , address=145.79.227.36:4517 , piece_id=I2XQTHHNCLL7JE4G3YZK57MLDAGKZFDRBXWJJ42MXBKFPVUEC24Q, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.743168583 (568h38m46.614251618s) [noise=true , address=145.79.227.14:4840 , piece_id=BX4IAN3OZKIVRXV4DDSVKH6TG74DWCYL7QEIZEFIFCT4L2R6EPRA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.757232152 (568h38m46.628315197s) [noise=true , address=145.79.227.5:4222 , piece_id=DNGIMZNDGXJCKXQNKOZJV7LQ7TER7YSOYHOCIKPCT3WHE2EBAZBQ, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:46.962745618 (568h38m46.833828663s) [noise=true , address=77.90.31.21:4880 , piece_id=3JGXN7SVIC4G3NQQOZL6NFNQ2V6XK2KINALDCUAYUG433NWZNUEQ, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:47.038069315 (568h38m46.90915236s) [noise=true , address=45.91.116.14:4189 , piece_id=6JYY6QKYDSRUDCZOT7NGTDMW2BGK3JHGVUUO34VQAZRSOOBYNXMA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: <nil> Dec 29 08:07:47.038088972 (568h38m46.909172007s) [seg_pos={0 0}, upload=498543]: Segment reached optimal threshold of 13 pieces. Dec 29 08:07:47.038321215 (568h38m46.90940429s) [noise=true , address=45.91.116.5:4298 , piece_id=KUE72SYO3YVC5QYUULTNNOUDX22UQXADXZQNL2GDOCJNHA4MZ7KA, seg_pos={0 0}, upload=498543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 08:07:47.038334320 (568h38m46.909417685s) [noise=true , address=45.91.116.5:4298 , piece_id=KUE72SYO3YVC5QYUULTNNOUDX22UQXADXZQNL2GDOCJNHA4MZ7KA, seg_pos={0 0}, upload=498543]: Long tail cancellation due to optimal threshold Dec 29 08:07:47.038446864 (568h38m46.909529899s) [noise=true , address=77.90.31.15:4700 , piece_id=4WGLCCL6SG5ZFGXPJMSOYMZVKFJUMXZGPRD75VI5DKNSLXVYVA2A, seg_pos={0 0}, upload=498543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:55792->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:55792->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55792->77.90.31.15:4700: use of closed network connection Dec 29 08:07:47.038478614 (568h38m46.909561669s) [noise=true , address=77.90.31.15:4700 , piece_id=4WGLCCL6SG5ZFGXPJMSOYMZVKFJUMXZGPRD75VI5DKNSLXVYVA2A, seg_pos={0 0}, upload=498543]: Long tail cancellation due to optimal threshold Dec 29 08:07:47.038820115 (568h38m46.90990317s) [noise=true , address=77.90.31.11:4555 , piece_id=W2B77TCGGJBPNQ5BPU3R23MBFQVGQ3JP3WIGTDPROLCLSTTCXDQQ, seg_pos={0 0}, upload=498543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 29 08:07:47.038831827 (568h38m46.909914872s) [noise=true , address=77.90.31.11:4555 , piece_id=W2B77TCGGJBPNQ5BPU3R23MBFQVGQ3JP3WIGTDPROLCLSTTCXDQQ, seg_pos={0 0}, upload=498543]: Long tail cancellation due to optimal threshold Dec 29 08:07:47.038860321 (568h38m46.909943366s) [seg_pos={0 0}, upload=498543]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:07:47.038954901 (568h38m46.910037936s) [upload=498543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:07:47.038960833 (568h38m46.910043878s) [upload=498543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:07:47.329690392 (568h38m47.200773447s) [upload=498543]: Done uploading segments... Dec 29 08:07:47.329714879 (568h38m47.200797924s) [upload=498543]: Upload finished. err: <nil> Duration: 30.002315242s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................X........................X..X......X.] duration: 4.443187373s trace: https://jaeger.dev.storj.io/trace/43163810b1d7c614 FAIL (2025-12-29 08:07:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.443210938s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..............X.X.X.............X..................] duration: 15.342262843s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:07:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 15.342277701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..............X.X.X.............X.................] duration: 25.70563055s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:06:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 25.705656488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................X........................X..X......X] duration: 5.001408018s trace: https://jaeger.dev.storj.io/trace/43163810b1d7c614 FAIL (2025-12-29 08:06:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001431141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.XX..X...X..XX..X...X.XXXX.X........X...X............XX....] duration: 5.990711735s trace: https://jaeger.dev.storj.io/trace/1560a75b678e2f49 FAIL (2025-12-29 08:06:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A) context deadline exceeded ======LOGS====== Dec 29 08:03:44.462140920 (568h34m44.333223965s) [upload=498482]: Starting upload Dec 29 08:03:44.462233697 (568h34m44.333316732s) [upload=498482]: Done starting upload Dec 29 08:03:44.462429038 (568h34m44.333512073s) [upload=498482]: Uploading segments... Dec 29 08:03:44.462473263 (568h34m44.333556308s) [upload=498482]: Got next segment. Inline: false Dec 29 08:03:44.462497729 (568h34m44.333580764s) [upload=498482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:03:44.462502389 (568h34m44.333585424s) [upload=498482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:03:44.710163052 (568h34m44.581246107s) [seg_pos={0 0}, upload=498482]: Begin upload segment... Dec 29 08:03:44.710223598 (568h34m44.581306643s) [seg_pos={0 0}, upload=498482]: Done begin upload segment. Dec 29 08:03:44.710260247 (568h34m44.581343292s) [noise=true , address=145.79.227.3:4150 , piece_id=MOENKMFUGZ5LFWS7C4XGQDNGZ5O2DMHR7QFPHDRPICVFZ2BVEUNQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.710273172 (568h34m44.581356217s) [noise=true , address=145.79.227.25:4619 , piece_id=HRU2HR4YN3GPZQKMWETIWJZSTFFCILJG6FDNSTXIYZVPNTRE5FPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711057676 (568h34m44.582140721s) [noise=true , address=145.79.227.41:4086 , piece_id=XX5V5TJP2SCIDPXJV6F44VEOZS24P7ZZBTI626TA3VDH5GCSGMCA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711201309 (568h34m44.582284354s) [noise=true , address=145.79.227.36:4518 , piece_id=WONGA3TCH2IHWXEKW625NVZ3W4DZ26SHJ6HDSKB4EINXDLDUS66Q, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711768479 (568h34m44.582851514s) [noise=true , address=145.79.227.16:4910 , piece_id=K2HEDXSRNGLT7N3RDDN6GNQ52KTPPKDXPWPCB3O7IXGQORIZ77AA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712100602 (568h34m44.583183647s) [noise=true , address=145.79.227.13:4802 , piece_id=HLDL63FTTFWHODP3FQFWIOWVPSXRJCX4NQR4GDHMGRNTOZ6A6EAA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712611253 (568h34m44.583694289s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712974756 (568h34m44.584057801s) [noise=true , address=145.79.227.9:4367 , piece_id=G672OOVZGKUMKIF4QDUDPLHBSJVT4GMZ66YG47L42AS3CARGSMPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713222257 (568h34m44.584305302s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713369968 (568h34m44.584453003s) [noise=true , address=145.79.227.6:4259 , piece_id=TFMVNJRCKCFC7LAEI64SAZGRLQCYZ2ISKFUYFS2QYWCLXLFDAFXA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713788886 (568h34m44.584871931s) [noise=true , address=145.79.227.26:4655 , piece_id=2GDHU54NLBZ7AXSEKA52CX3ZONGMIQEQEBPVDLKNYHAK2IHSREDQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713906760 (568h34m44.584989795s) [noise=true , address=77.90.31.21:4878 , piece_id=CLZWJVQJMDT55UJH2TBQFDNT36EOBOFMUXDTHDIMWJCQ2QUN4F6A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.714447280 (568h34m44.585530325s) [noise=true , address=77.90.31.13:4627 , piece_id=WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.714974954 (568h34m44.586057989s) [noise=true , address=145.79.227.39:4015 , piece_id=W72SG2I4N2ULNXDJVXS4WG76KJR4QZX62EKGBXKZIU5O7UWTGUIQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.715780939 (568h34m44.586863984s) [noise=true , address=45.91.116.10:4082 , piece_id=HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.716360433 (568h34m44.587443478s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:46.318343034 (568h34m46.189426079s) [upload=498482]: Next returned nil segment Dec 29 08:03:46.318408128 (568h34m46.189491173s) [upload=498482]: Waiting for error group managing segments... Dec 29 08:03:46.702277613 (568h34m46.573360648s) [noise=true , address=145.79.227.16:4910 , piece_id=K2HEDXSRNGLT7N3RDDN6GNQ52KTPPKDXPWPCB3O7IXGQORIZ77AA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.704282191 (568h34m46.575365226s) [noise=true , address=145.79.227.41:4086 , piece_id=XX5V5TJP2SCIDPXJV6F44VEOZS24P7ZZBTI626TA3VDH5GCSGMCA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.704737437 (568h34m46.575820472s) [noise=true , address=145.79.227.26:4655 , piece_id=2GDHU54NLBZ7AXSEKA52CX3ZONGMIQEQEBPVDLKNYHAK2IHSREDQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.706696477 (568h34m46.577779512s) [noise=true , address=145.79.227.6:4259 , piece_id=TFMVNJRCKCFC7LAEI64SAZGRLQCYZ2ISKFUYFS2QYWCLXLFDAFXA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.707664140 (568h34m46.578747185s) [noise=true , address=145.79.227.25:4619 , piece_id=HRU2HR4YN3GPZQKMWETIWJZSTFFCILJG6FDNSTXIYZVPNTRE5FPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.708282899 (568h34m46.579365944s) [noise=true , address=145.79.227.9:4367 , piece_id=G672OOVZGKUMKIF4QDUDPLHBSJVT4GMZ66YG47L42AS3CARGSMPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.708886598 (568h34m46.579969643s) [noise=true , address=145.79.227.39:4015 , piece_id=W72SG2I4N2ULNXDJVXS4WG76KJR4QZX62EKGBXKZIU5O7UWTGUIQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.716454498 (568h34m46.587537543s) [noise=true , address=145.79.227.3:4150 , piece_id=MOENKMFUGZ5LFWS7C4XGQDNGZ5O2DMHR7QFPHDRPICVFZ2BVEUNQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.719581463 (568h34m46.590664508s) [noise=true , address=145.79.227.13:4802 , piece_id=HLDL63FTTFWHODP3FQFWIOWVPSXRJCX4NQR4GDHMGRNTOZ6A6EAA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.719901993 (568h34m46.590985028s) [noise=true , address=145.79.227.36:4518 , piece_id=WONGA3TCH2IHWXEKW625NVZ3W4DZ26SHJ6HDSKB4EINXDLDUS66Q, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.959555730 (568h34m46.830638775s) [noise=true , address=77.90.31.13:4627 , piece_id=WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.000861934 (568h34m46.871944989s) [noise=true , address=77.90.31.21:4878 , piece_id=CLZWJVQJMDT55UJH2TBQFDNT36EOBOFMUXDTHDIMWJCQ2QUN4F6A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.031877602 (568h34m46.902960647s) [noise=true , address=45.91.116.10:4082 , piece_id=HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.031903672 (568h34m46.902986707s) [seg_pos={0 0}, upload=498482]: Segment reached optimal threshold of 13 pieces. Dec 29 08:03:47.032153888 (568h34m46.903236943s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 08:03:47.032162564 (568h34m46.90324562s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032232157 (568h34m46.903315192s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 08:03:47.032248058 (568h34m46.903331093s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032460803 (568h34m46.903543838s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:58436->77.90.31.19:4841: use of closed network connection; context canceled Dec 29 08:03:47.032467906 (568h34m46.903550951s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032482934 (568h34m46.903565979s) [seg_pos={0 0}, upload=498482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:03:47.032557096 (568h34m46.903640141s) [upload=498482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:03:47.032561665 (568h34m46.90364471s) [upload=498482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:03:47.349046430 (568h34m47.220129465s) [upload=498482]: Done uploading segments... Dec 29 08:03:47.349078150 (568h34m47.220161196s) [upload=498482]: Upload finished. err: <nil> Duration: 5.990754868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X..............X.X.X.............X................] duration: 15.398859949s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:06:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 15.398877352s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X........................X..X......] duration: 3.750960203s trace: https://jaeger.dev.storj.io/trace/361d49667c5f8ac6 FAIL (2025-12-29 08:06:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.750990701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.XX..X...X..XX..X...X.XXXX.X........X...X............XX...] duration: 5.738185847s trace: https://jaeger.dev.storj.io/trace/1560a75b678e2f49 FAIL (2025-12-29 08:06:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A) context deadline exceeded ======LOGS====== Dec 29 08:03:44.462140920 (568h34m44.333223965s) [upload=498482]: Starting upload Dec 29 08:03:44.462233697 (568h34m44.333316732s) [upload=498482]: Done starting upload Dec 29 08:03:44.462429038 (568h34m44.333512073s) [upload=498482]: Uploading segments... Dec 29 08:03:44.462473263 (568h34m44.333556308s) [upload=498482]: Got next segment. Inline: false Dec 29 08:03:44.462497729 (568h34m44.333580764s) [upload=498482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:03:44.462502389 (568h34m44.333585424s) [upload=498482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:03:44.710163052 (568h34m44.581246107s) [seg_pos={0 0}, upload=498482]: Begin upload segment... Dec 29 08:03:44.710223598 (568h34m44.581306643s) [seg_pos={0 0}, upload=498482]: Done begin upload segment. Dec 29 08:03:44.710260247 (568h34m44.581343292s) [noise=true , address=145.79.227.3:4150 , piece_id=MOENKMFUGZ5LFWS7C4XGQDNGZ5O2DMHR7QFPHDRPICVFZ2BVEUNQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.710273172 (568h34m44.581356217s) [noise=true , address=145.79.227.25:4619 , piece_id=HRU2HR4YN3GPZQKMWETIWJZSTFFCILJG6FDNSTXIYZVPNTRE5FPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711057676 (568h34m44.582140721s) [noise=true , address=145.79.227.41:4086 , piece_id=XX5V5TJP2SCIDPXJV6F44VEOZS24P7ZZBTI626TA3VDH5GCSGMCA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711201309 (568h34m44.582284354s) [noise=true , address=145.79.227.36:4518 , piece_id=WONGA3TCH2IHWXEKW625NVZ3W4DZ26SHJ6HDSKB4EINXDLDUS66Q, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711768479 (568h34m44.582851514s) [noise=true , address=145.79.227.16:4910 , piece_id=K2HEDXSRNGLT7N3RDDN6GNQ52KTPPKDXPWPCB3O7IXGQORIZ77AA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712100602 (568h34m44.583183647s) [noise=true , address=145.79.227.13:4802 , piece_id=HLDL63FTTFWHODP3FQFWIOWVPSXRJCX4NQR4GDHMGRNTOZ6A6EAA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712611253 (568h34m44.583694289s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712974756 (568h34m44.584057801s) [noise=true , address=145.79.227.9:4367 , piece_id=G672OOVZGKUMKIF4QDUDPLHBSJVT4GMZ66YG47L42AS3CARGSMPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713222257 (568h34m44.584305302s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713369968 (568h34m44.584453003s) [noise=true , address=145.79.227.6:4259 , piece_id=TFMVNJRCKCFC7LAEI64SAZGRLQCYZ2ISKFUYFS2QYWCLXLFDAFXA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713788886 (568h34m44.584871931s) [noise=true , address=145.79.227.26:4655 , piece_id=2GDHU54NLBZ7AXSEKA52CX3ZONGMIQEQEBPVDLKNYHAK2IHSREDQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713906760 (568h34m44.584989795s) [noise=true , address=77.90.31.21:4878 , piece_id=CLZWJVQJMDT55UJH2TBQFDNT36EOBOFMUXDTHDIMWJCQ2QUN4F6A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.714447280 (568h34m44.585530325s) [noise=true , address=77.90.31.13:4627 , piece_id=WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.714974954 (568h34m44.586057989s) [noise=true , address=145.79.227.39:4015 , piece_id=W72SG2I4N2ULNXDJVXS4WG76KJR4QZX62EKGBXKZIU5O7UWTGUIQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.715780939 (568h34m44.586863984s) [noise=true , address=45.91.116.10:4082 , piece_id=HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.716360433 (568h34m44.587443478s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:46.318343034 (568h34m46.189426079s) [upload=498482]: Next returned nil segment Dec 29 08:03:46.318408128 (568h34m46.189491173s) [upload=498482]: Waiting for error group managing segments... Dec 29 08:03:46.702277613 (568h34m46.573360648s) [noise=true , address=145.79.227.16:4910 , piece_id=K2HEDXSRNGLT7N3RDDN6GNQ52KTPPKDXPWPCB3O7IXGQORIZ77AA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.704282191 (568h34m46.575365226s) [noise=true , address=145.79.227.41:4086 , piece_id=XX5V5TJP2SCIDPXJV6F44VEOZS24P7ZZBTI626TA3VDH5GCSGMCA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.704737437 (568h34m46.575820472s) [noise=true , address=145.79.227.26:4655 , piece_id=2GDHU54NLBZ7AXSEKA52CX3ZONGMIQEQEBPVDLKNYHAK2IHSREDQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.706696477 (568h34m46.577779512s) [noise=true , address=145.79.227.6:4259 , piece_id=TFMVNJRCKCFC7LAEI64SAZGRLQCYZ2ISKFUYFS2QYWCLXLFDAFXA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.707664140 (568h34m46.578747185s) [noise=true , address=145.79.227.25:4619 , piece_id=HRU2HR4YN3GPZQKMWETIWJZSTFFCILJG6FDNSTXIYZVPNTRE5FPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.708282899 (568h34m46.579365944s) [noise=true , address=145.79.227.9:4367 , piece_id=G672OOVZGKUMKIF4QDUDPLHBSJVT4GMZ66YG47L42AS3CARGSMPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.708886598 (568h34m46.579969643s) [noise=true , address=145.79.227.39:4015 , piece_id=W72SG2I4N2ULNXDJVXS4WG76KJR4QZX62EKGBXKZIU5O7UWTGUIQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.716454498 (568h34m46.587537543s) [noise=true , address=145.79.227.3:4150 , piece_id=MOENKMFUGZ5LFWS7C4XGQDNGZ5O2DMHR7QFPHDRPICVFZ2BVEUNQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.719581463 (568h34m46.590664508s) [noise=true , address=145.79.227.13:4802 , piece_id=HLDL63FTTFWHODP3FQFWIOWVPSXRJCX4NQR4GDHMGRNTOZ6A6EAA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.719901993 (568h34m46.590985028s) [noise=true , address=145.79.227.36:4518 , piece_id=WONGA3TCH2IHWXEKW625NVZ3W4DZ26SHJ6HDSKB4EINXDLDUS66Q, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.959555730 (568h34m46.830638775s) [noise=true , address=77.90.31.13:4627 , piece_id=WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.000861934 (568h34m46.871944989s) [noise=true , address=77.90.31.21:4878 , piece_id=CLZWJVQJMDT55UJH2TBQFDNT36EOBOFMUXDTHDIMWJCQ2QUN4F6A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.031877602 (568h34m46.902960647s) [noise=true , address=45.91.116.10:4082 , piece_id=HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.031903672 (568h34m46.902986707s) [seg_pos={0 0}, upload=498482]: Segment reached optimal threshold of 13 pieces. Dec 29 08:03:47.032153888 (568h34m46.903236943s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 08:03:47.032162564 (568h34m46.90324562s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032232157 (568h34m46.903315192s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 08:03:47.032248058 (568h34m46.903331093s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032460803 (568h34m46.903543838s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:58436->77.90.31.19:4841: use of closed network connection; context canceled Dec 29 08:03:47.032467906 (568h34m46.903550951s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032482934 (568h34m46.903565979s) [seg_pos={0 0}, upload=498482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:03:47.032557096 (568h34m46.903640141s) [upload=498482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:03:47.032561665 (568h34m46.90364471s) [upload=498482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:03:47.349046430 (568h34m47.220129465s) [upload=498482]: Done uploading segments... Dec 29 08:03:47.349078150 (568h34m47.220161196s) [upload=498482]: Upload finished. err: <nil> Duration: 5.738236443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X..............X.X.X.............X...............] duration: 26.159398291s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:06:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 26.159418719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X........................X..X.....] duration: 4.369115095s trace: https://jaeger.dev.storj.io/trace/361d49667c5f8ac6 FAIL (2025-12-29 08:05:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.369145572s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................X........................X..X....] duration: 4.758463562s trace: https://jaeger.dev.storj.io/trace/361d49667c5f8ac6 FAIL (2025-12-29 08:05:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.758482248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.XX..X...X..XX..X...X.XXXX.X........X...X............XX..] duration: 5.932655793s trace: https://jaeger.dev.storj.io/trace/1560a75b678e2f49 FAIL (2025-12-29 08:05:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A) context deadline exceeded ======LOGS====== Dec 29 08:03:44.462140920 (568h34m44.333223965s) [upload=498482]: Starting upload Dec 29 08:03:44.462233697 (568h34m44.333316732s) [upload=498482]: Done starting upload Dec 29 08:03:44.462429038 (568h34m44.333512073s) [upload=498482]: Uploading segments... Dec 29 08:03:44.462473263 (568h34m44.333556308s) [upload=498482]: Got next segment. Inline: false Dec 29 08:03:44.462497729 (568h34m44.333580764s) [upload=498482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:03:44.462502389 (568h34m44.333585424s) [upload=498482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:03:44.710163052 (568h34m44.581246107s) [seg_pos={0 0}, upload=498482]: Begin upload segment... Dec 29 08:03:44.710223598 (568h34m44.581306643s) [seg_pos={0 0}, upload=498482]: Done begin upload segment. Dec 29 08:03:44.710260247 (568h34m44.581343292s) [noise=true , address=145.79.227.3:4150 , piece_id=MOENKMFUGZ5LFWS7C4XGQDNGZ5O2DMHR7QFPHDRPICVFZ2BVEUNQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.710273172 (568h34m44.581356217s) [noise=true , address=145.79.227.25:4619 , piece_id=HRU2HR4YN3GPZQKMWETIWJZSTFFCILJG6FDNSTXIYZVPNTRE5FPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711057676 (568h34m44.582140721s) [noise=true , address=145.79.227.41:4086 , piece_id=XX5V5TJP2SCIDPXJV6F44VEOZS24P7ZZBTI626TA3VDH5GCSGMCA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711201309 (568h34m44.582284354s) [noise=true , address=145.79.227.36:4518 , piece_id=WONGA3TCH2IHWXEKW625NVZ3W4DZ26SHJ6HDSKB4EINXDLDUS66Q, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711768479 (568h34m44.582851514s) [noise=true , address=145.79.227.16:4910 , piece_id=K2HEDXSRNGLT7N3RDDN6GNQ52KTPPKDXPWPCB3O7IXGQORIZ77AA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712100602 (568h34m44.583183647s) [noise=true , address=145.79.227.13:4802 , piece_id=HLDL63FTTFWHODP3FQFWIOWVPSXRJCX4NQR4GDHMGRNTOZ6A6EAA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712611253 (568h34m44.583694289s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712974756 (568h34m44.584057801s) [noise=true , address=145.79.227.9:4367 , piece_id=G672OOVZGKUMKIF4QDUDPLHBSJVT4GMZ66YG47L42AS3CARGSMPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713222257 (568h34m44.584305302s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713369968 (568h34m44.584453003s) [noise=true , address=145.79.227.6:4259 , piece_id=TFMVNJRCKCFC7LAEI64SAZGRLQCYZ2ISKFUYFS2QYWCLXLFDAFXA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713788886 (568h34m44.584871931s) [noise=true , address=145.79.227.26:4655 , piece_id=2GDHU54NLBZ7AXSEKA52CX3ZONGMIQEQEBPVDLKNYHAK2IHSREDQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713906760 (568h34m44.584989795s) [noise=true , address=77.90.31.21:4878 , piece_id=CLZWJVQJMDT55UJH2TBQFDNT36EOBOFMUXDTHDIMWJCQ2QUN4F6A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.714447280 (568h34m44.585530325s) [noise=true , address=77.90.31.13:4627 , piece_id=WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.714974954 (568h34m44.586057989s) [noise=true , address=145.79.227.39:4015 , piece_id=W72SG2I4N2ULNXDJVXS4WG76KJR4QZX62EKGBXKZIU5O7UWTGUIQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.715780939 (568h34m44.586863984s) [noise=true , address=45.91.116.10:4082 , piece_id=HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.716360433 (568h34m44.587443478s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:46.318343034 (568h34m46.189426079s) [upload=498482]: Next returned nil segment Dec 29 08:03:46.318408128 (568h34m46.189491173s) [upload=498482]: Waiting for error group managing segments... Dec 29 08:03:46.702277613 (568h34m46.573360648s) [noise=true , address=145.79.227.16:4910 , piece_id=K2HEDXSRNGLT7N3RDDN6GNQ52KTPPKDXPWPCB3O7IXGQORIZ77AA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.704282191 (568h34m46.575365226s) [noise=true , address=145.79.227.41:4086 , piece_id=XX5V5TJP2SCIDPXJV6F44VEOZS24P7ZZBTI626TA3VDH5GCSGMCA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.704737437 (568h34m46.575820472s) [noise=true , address=145.79.227.26:4655 , piece_id=2GDHU54NLBZ7AXSEKA52CX3ZONGMIQEQEBPVDLKNYHAK2IHSREDQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.706696477 (568h34m46.577779512s) [noise=true , address=145.79.227.6:4259 , piece_id=TFMVNJRCKCFC7LAEI64SAZGRLQCYZ2ISKFUYFS2QYWCLXLFDAFXA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.707664140 (568h34m46.578747185s) [noise=true , address=145.79.227.25:4619 , piece_id=HRU2HR4YN3GPZQKMWETIWJZSTFFCILJG6FDNSTXIYZVPNTRE5FPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.708282899 (568h34m46.579365944s) [noise=true , address=145.79.227.9:4367 , piece_id=G672OOVZGKUMKIF4QDUDPLHBSJVT4GMZ66YG47L42AS3CARGSMPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.708886598 (568h34m46.579969643s) [noise=true , address=145.79.227.39:4015 , piece_id=W72SG2I4N2ULNXDJVXS4WG76KJR4QZX62EKGBXKZIU5O7UWTGUIQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.716454498 (568h34m46.587537543s) [noise=true , address=145.79.227.3:4150 , piece_id=MOENKMFUGZ5LFWS7C4XGQDNGZ5O2DMHR7QFPHDRPICVFZ2BVEUNQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.719581463 (568h34m46.590664508s) [noise=true , address=145.79.227.13:4802 , piece_id=HLDL63FTTFWHODP3FQFWIOWVPSXRJCX4NQR4GDHMGRNTOZ6A6EAA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.719901993 (568h34m46.590985028s) [noise=true , address=145.79.227.36:4518 , piece_id=WONGA3TCH2IHWXEKW625NVZ3W4DZ26SHJ6HDSKB4EINXDLDUS66Q, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.959555730 (568h34m46.830638775s) [noise=true , address=77.90.31.13:4627 , piece_id=WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.000861934 (568h34m46.871944989s) [noise=true , address=77.90.31.21:4878 , piece_id=CLZWJVQJMDT55UJH2TBQFDNT36EOBOFMUXDTHDIMWJCQ2QUN4F6A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.031877602 (568h34m46.902960647s) [noise=true , address=45.91.116.10:4082 , piece_id=HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.031903672 (568h34m46.902986707s) [seg_pos={0 0}, upload=498482]: Segment reached optimal threshold of 13 pieces. Dec 29 08:03:47.032153888 (568h34m46.903236943s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 08:03:47.032162564 (568h34m46.90324562s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032232157 (568h34m46.903315192s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 08:03:47.032248058 (568h34m46.903331093s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032460803 (568h34m46.903543838s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:58436->77.90.31.19:4841: use of closed network connection; context canceled Dec 29 08:03:47.032467906 (568h34m46.903550951s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032482934 (568h34m46.903565979s) [seg_pos={0 0}, upload=498482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:03:47.032557096 (568h34m46.903640141s) [upload=498482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:03:47.032561665 (568h34m46.90364471s) [upload=498482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:03:47.349046430 (568h34m47.220129465s) [upload=498482]: Done uploading segments... Dec 29 08:03:47.349078150 (568h34m47.220161196s) [upload=498482]: Upload finished. err: <nil> Duration: 5.932687523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X..............X.X.X.............X..............] duration: 35.517845155s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:05:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 35.517873248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................X........................X..X...] duration: 3.505162356s trace: https://jaeger.dev.storj.io/trace/361d49667c5f8ac6 FAIL (2025-12-29 08:05:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505194307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.XX..X...X..XX..X...X.XXXX.X........X...X............XX.] duration: 6.03421761s trace: https://jaeger.dev.storj.io/trace/1560a75b678e2f49 FAIL (2025-12-29 08:04:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A) context deadline exceeded ======LOGS====== Dec 29 08:03:44.462140920 (568h34m44.333223965s) [upload=498482]: Starting upload Dec 29 08:03:44.462233697 (568h34m44.333316732s) [upload=498482]: Done starting upload Dec 29 08:03:44.462429038 (568h34m44.333512073s) [upload=498482]: Uploading segments... Dec 29 08:03:44.462473263 (568h34m44.333556308s) [upload=498482]: Got next segment. Inline: false Dec 29 08:03:44.462497729 (568h34m44.333580764s) [upload=498482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:03:44.462502389 (568h34m44.333585424s) [upload=498482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:03:44.710163052 (568h34m44.581246107s) [seg_pos={0 0}, upload=498482]: Begin upload segment... Dec 29 08:03:44.710223598 (568h34m44.581306643s) [seg_pos={0 0}, upload=498482]: Done begin upload segment. Dec 29 08:03:44.710260247 (568h34m44.581343292s) [noise=true , address=145.79.227.3:4150 , piece_id=MOENKMFUGZ5LFWS7C4XGQDNGZ5O2DMHR7QFPHDRPICVFZ2BVEUNQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.710273172 (568h34m44.581356217s) [noise=true , address=145.79.227.25:4619 , piece_id=HRU2HR4YN3GPZQKMWETIWJZSTFFCILJG6FDNSTXIYZVPNTRE5FPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711057676 (568h34m44.582140721s) [noise=true , address=145.79.227.41:4086 , piece_id=XX5V5TJP2SCIDPXJV6F44VEOZS24P7ZZBTI626TA3VDH5GCSGMCA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711201309 (568h34m44.582284354s) [noise=true , address=145.79.227.36:4518 , piece_id=WONGA3TCH2IHWXEKW625NVZ3W4DZ26SHJ6HDSKB4EINXDLDUS66Q, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711768479 (568h34m44.582851514s) [noise=true , address=145.79.227.16:4910 , piece_id=K2HEDXSRNGLT7N3RDDN6GNQ52KTPPKDXPWPCB3O7IXGQORIZ77AA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712100602 (568h34m44.583183647s) [noise=true , address=145.79.227.13:4802 , piece_id=HLDL63FTTFWHODP3FQFWIOWVPSXRJCX4NQR4GDHMGRNTOZ6A6EAA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712611253 (568h34m44.583694289s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712974756 (568h34m44.584057801s) [noise=true , address=145.79.227.9:4367 , piece_id=G672OOVZGKUMKIF4QDUDPLHBSJVT4GMZ66YG47L42AS3CARGSMPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713222257 (568h34m44.584305302s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713369968 (568h34m44.584453003s) [noise=true , address=145.79.227.6:4259 , piece_id=TFMVNJRCKCFC7LAEI64SAZGRLQCYZ2ISKFUYFS2QYWCLXLFDAFXA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713788886 (568h34m44.584871931s) [noise=true , address=145.79.227.26:4655 , piece_id=2GDHU54NLBZ7AXSEKA52CX3ZONGMIQEQEBPVDLKNYHAK2IHSREDQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713906760 (568h34m44.584989795s) [noise=true , address=77.90.31.21:4878 , piece_id=CLZWJVQJMDT55UJH2TBQFDNT36EOBOFMUXDTHDIMWJCQ2QUN4F6A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.714447280 (568h34m44.585530325s) [noise=true , address=77.90.31.13:4627 , piece_id=WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.714974954 (568h34m44.586057989s) [noise=true , address=145.79.227.39:4015 , piece_id=W72SG2I4N2ULNXDJVXS4WG76KJR4QZX62EKGBXKZIU5O7UWTGUIQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.715780939 (568h34m44.586863984s) [noise=true , address=45.91.116.10:4082 , piece_id=HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.716360433 (568h34m44.587443478s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:46.318343034 (568h34m46.189426079s) [upload=498482]: Next returned nil segment Dec 29 08:03:46.318408128 (568h34m46.189491173s) [upload=498482]: Waiting for error group managing segments... Dec 29 08:03:46.702277613 (568h34m46.573360648s) [noise=true , address=145.79.227.16:4910 , piece_id=K2HEDXSRNGLT7N3RDDN6GNQ52KTPPKDXPWPCB3O7IXGQORIZ77AA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.704282191 (568h34m46.575365226s) [noise=true , address=145.79.227.41:4086 , piece_id=XX5V5TJP2SCIDPXJV6F44VEOZS24P7ZZBTI626TA3VDH5GCSGMCA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.704737437 (568h34m46.575820472s) [noise=true , address=145.79.227.26:4655 , piece_id=2GDHU54NLBZ7AXSEKA52CX3ZONGMIQEQEBPVDLKNYHAK2IHSREDQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.706696477 (568h34m46.577779512s) [noise=true , address=145.79.227.6:4259 , piece_id=TFMVNJRCKCFC7LAEI64SAZGRLQCYZ2ISKFUYFS2QYWCLXLFDAFXA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.707664140 (568h34m46.578747185s) [noise=true , address=145.79.227.25:4619 , piece_id=HRU2HR4YN3GPZQKMWETIWJZSTFFCILJG6FDNSTXIYZVPNTRE5FPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.708282899 (568h34m46.579365944s) [noise=true , address=145.79.227.9:4367 , piece_id=G672OOVZGKUMKIF4QDUDPLHBSJVT4GMZ66YG47L42AS3CARGSMPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.708886598 (568h34m46.579969643s) [noise=true , address=145.79.227.39:4015 , piece_id=W72SG2I4N2ULNXDJVXS4WG76KJR4QZX62EKGBXKZIU5O7UWTGUIQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.716454498 (568h34m46.587537543s) [noise=true , address=145.79.227.3:4150 , piece_id=MOENKMFUGZ5LFWS7C4XGQDNGZ5O2DMHR7QFPHDRPICVFZ2BVEUNQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.719581463 (568h34m46.590664508s) [noise=true , address=145.79.227.13:4802 , piece_id=HLDL63FTTFWHODP3FQFWIOWVPSXRJCX4NQR4GDHMGRNTOZ6A6EAA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.719901993 (568h34m46.590985028s) [noise=true , address=145.79.227.36:4518 , piece_id=WONGA3TCH2IHWXEKW625NVZ3W4DZ26SHJ6HDSKB4EINXDLDUS66Q, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.959555730 (568h34m46.830638775s) [noise=true , address=77.90.31.13:4627 , piece_id=WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.000861934 (568h34m46.871944989s) [noise=true , address=77.90.31.21:4878 , piece_id=CLZWJVQJMDT55UJH2TBQFDNT36EOBOFMUXDTHDIMWJCQ2QUN4F6A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.031877602 (568h34m46.902960647s) [noise=true , address=45.91.116.10:4082 , piece_id=HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.031903672 (568h34m46.902986707s) [seg_pos={0 0}, upload=498482]: Segment reached optimal threshold of 13 pieces. Dec 29 08:03:47.032153888 (568h34m46.903236943s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 08:03:47.032162564 (568h34m46.90324562s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032232157 (568h34m46.903315192s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 08:03:47.032248058 (568h34m46.903331093s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032460803 (568h34m46.903543838s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:58436->77.90.31.19:4841: use of closed network connection; context canceled Dec 29 08:03:47.032467906 (568h34m46.903550951s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032482934 (568h34m46.903565979s) [seg_pos={0 0}, upload=498482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:03:47.032557096 (568h34m46.903640141s) [upload=498482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:03:47.032561665 (568h34m46.90364471s) [upload=498482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:03:47.349046430 (568h34m47.220129465s) [upload=498482]: Done uploading segments... Dec 29 08:03:47.349078150 (568h34m47.220161196s) [upload=498482]: Upload finished. err: <nil> Duration: 6.034244071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X..............X.X.X.............X.............] duration: 41.464117336s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:04:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 41.464135039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................X........................X..X..] duration: 3.476976095s trace: https://jaeger.dev.storj.io/trace/361d49667c5f8ac6 FAIL (2025-12-29 08:04:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.477006874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X.XX..X...X..XX..X...X.XXXX.X........X...X............XX] duration: 30.003000918s trace: https://jaeger.dev.storj.io/trace/1560a75b678e2f49 FAIL (2025-12-29 08:04:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A) context deadline exceeded ======LOGS====== Dec 29 08:03:44.462140920 (568h34m44.333223965s) [upload=498482]: Starting upload Dec 29 08:03:44.462233697 (568h34m44.333316732s) [upload=498482]: Done starting upload Dec 29 08:03:44.462429038 (568h34m44.333512073s) [upload=498482]: Uploading segments... Dec 29 08:03:44.462473263 (568h34m44.333556308s) [upload=498482]: Got next segment. Inline: false Dec 29 08:03:44.462497729 (568h34m44.333580764s) [upload=498482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:03:44.462502389 (568h34m44.333585424s) [upload=498482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:03:44.710163052 (568h34m44.581246107s) [seg_pos={0 0}, upload=498482]: Begin upload segment... Dec 29 08:03:44.710223598 (568h34m44.581306643s) [seg_pos={0 0}, upload=498482]: Done begin upload segment. Dec 29 08:03:44.710260247 (568h34m44.581343292s) [noise=true , address=145.79.227.3:4150 , piece_id=MOENKMFUGZ5LFWS7C4XGQDNGZ5O2DMHR7QFPHDRPICVFZ2BVEUNQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.710273172 (568h34m44.581356217s) [noise=true , address=145.79.227.25:4619 , piece_id=HRU2HR4YN3GPZQKMWETIWJZSTFFCILJG6FDNSTXIYZVPNTRE5FPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711057676 (568h34m44.582140721s) [noise=true , address=145.79.227.41:4086 , piece_id=XX5V5TJP2SCIDPXJV6F44VEOZS24P7ZZBTI626TA3VDH5GCSGMCA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711201309 (568h34m44.582284354s) [noise=true , address=145.79.227.36:4518 , piece_id=WONGA3TCH2IHWXEKW625NVZ3W4DZ26SHJ6HDSKB4EINXDLDUS66Q, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.711768479 (568h34m44.582851514s) [noise=true , address=145.79.227.16:4910 , piece_id=K2HEDXSRNGLT7N3RDDN6GNQ52KTPPKDXPWPCB3O7IXGQORIZ77AA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712100602 (568h34m44.583183647s) [noise=true , address=145.79.227.13:4802 , piece_id=HLDL63FTTFWHODP3FQFWIOWVPSXRJCX4NQR4GDHMGRNTOZ6A6EAA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712611253 (568h34m44.583694289s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.712974756 (568h34m44.584057801s) [noise=true , address=145.79.227.9:4367 , piece_id=G672OOVZGKUMKIF4QDUDPLHBSJVT4GMZ66YG47L42AS3CARGSMPA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713222257 (568h34m44.584305302s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713369968 (568h34m44.584453003s) [noise=true , address=145.79.227.6:4259 , piece_id=TFMVNJRCKCFC7LAEI64SAZGRLQCYZ2ISKFUYFS2QYWCLXLFDAFXA, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713788886 (568h34m44.584871931s) [noise=true , address=145.79.227.26:4655 , piece_id=2GDHU54NLBZ7AXSEKA52CX3ZONGMIQEQEBPVDLKNYHAK2IHSREDQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.713906760 (568h34m44.584989795s) [noise=true , address=77.90.31.21:4878 , piece_id=CLZWJVQJMDT55UJH2TBQFDNT36EOBOFMUXDTHDIMWJCQ2QUN4F6A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.714447280 (568h34m44.585530325s) [noise=true , address=77.90.31.13:4627 , piece_id=WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.714974954 (568h34m44.586057989s) [noise=true , address=145.79.227.39:4015 , piece_id=W72SG2I4N2ULNXDJVXS4WG76KJR4QZX62EKGBXKZIU5O7UWTGUIQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.715780939 (568h34m44.586863984s) [noise=true , address=45.91.116.10:4082 , piece_id=HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:44.716360433 (568h34m44.587443478s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Uploading piece... Dec 29 08:03:46.318343034 (568h34m46.189426079s) [upload=498482]: Next returned nil segment Dec 29 08:03:46.318408128 (568h34m46.189491173s) [upload=498482]: Waiting for error group managing segments... Dec 29 08:03:46.702277613 (568h34m46.573360648s) [noise=true , address=145.79.227.16:4910 , piece_id=K2HEDXSRNGLT7N3RDDN6GNQ52KTPPKDXPWPCB3O7IXGQORIZ77AA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.704282191 (568h34m46.575365226s) [noise=true , address=145.79.227.41:4086 , piece_id=XX5V5TJP2SCIDPXJV6F44VEOZS24P7ZZBTI626TA3VDH5GCSGMCA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.704737437 (568h34m46.575820472s) [noise=true , address=145.79.227.26:4655 , piece_id=2GDHU54NLBZ7AXSEKA52CX3ZONGMIQEQEBPVDLKNYHAK2IHSREDQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.706696477 (568h34m46.577779512s) [noise=true , address=145.79.227.6:4259 , piece_id=TFMVNJRCKCFC7LAEI64SAZGRLQCYZ2ISKFUYFS2QYWCLXLFDAFXA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.707664140 (568h34m46.578747185s) [noise=true , address=145.79.227.25:4619 , piece_id=HRU2HR4YN3GPZQKMWETIWJZSTFFCILJG6FDNSTXIYZVPNTRE5FPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.708282899 (568h34m46.579365944s) [noise=true , address=145.79.227.9:4367 , piece_id=G672OOVZGKUMKIF4QDUDPLHBSJVT4GMZ66YG47L42AS3CARGSMPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.708886598 (568h34m46.579969643s) [noise=true , address=145.79.227.39:4015 , piece_id=W72SG2I4N2ULNXDJVXS4WG76KJR4QZX62EKGBXKZIU5O7UWTGUIQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.716454498 (568h34m46.587537543s) [noise=true , address=145.79.227.3:4150 , piece_id=MOENKMFUGZ5LFWS7C4XGQDNGZ5O2DMHR7QFPHDRPICVFZ2BVEUNQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.719581463 (568h34m46.590664508s) [noise=true , address=145.79.227.13:4802 , piece_id=HLDL63FTTFWHODP3FQFWIOWVPSXRJCX4NQR4GDHMGRNTOZ6A6EAA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.719901993 (568h34m46.590985028s) [noise=true , address=145.79.227.36:4518 , piece_id=WONGA3TCH2IHWXEKW625NVZ3W4DZ26SHJ6HDSKB4EINXDLDUS66Q, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:46.959555730 (568h34m46.830638775s) [noise=true , address=77.90.31.13:4627 , piece_id=WYTUAPBBZ7XFXN5AYPSCJ2SR4GBEZRDMVYK6TZMS6BAN745PMOMQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.000861934 (568h34m46.871944989s) [noise=true , address=77.90.31.21:4878 , piece_id=CLZWJVQJMDT55UJH2TBQFDNT36EOBOFMUXDTHDIMWJCQ2QUN4F6A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.031877602 (568h34m46.902960647s) [noise=true , address=45.91.116.10:4082 , piece_id=HVKMFU2LPF7YV4HLHFWI7TCB6W56SCAQHT4FFXMXXVZQT2FXDJ6A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: <nil> Dec 29 08:03:47.031903672 (568h34m46.902986707s) [seg_pos={0 0}, upload=498482]: Segment reached optimal threshold of 13 pieces. Dec 29 08:03:47.032153888 (568h34m46.903236943s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 08:03:47.032162564 (568h34m46.90324562s) [noise=true , address=45.91.116.9:4263 , piece_id=BUVDCPIM2PYHMDFCY5GMJ5MY5PRFAQYTGGSUBNFLJPQS5I3VC4FQ, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032232157 (568h34m46.903315192s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 08:03:47.032248058 (568h34m46.903331093s) [noise=true , address=45.91.116.6:4975 , piece_id=JLNGJLWAMV5JDP4SGWCTRFS56RDEKYPXDJPABD3S2DMIMQOJCKPA, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032460803 (568h34m46.903543838s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:58436->77.90.31.19:4841: use of closed network connection; context canceled Dec 29 08:03:47.032467906 (568h34m46.903550951s) [noise=true , address=77.90.31.19:4841 , piece_id=PMQ6SGLDAT42TJZRMBHVAIKVLCIITIMXGYXS6KLPAKXGJHE3EX4A, seg_pos={0 0}, upload=498482]: Long tail cancellation due to optimal threshold Dec 29 08:03:47.032482934 (568h34m46.903565979s) [seg_pos={0 0}, upload=498482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:03:47.032557096 (568h34m46.903640141s) [upload=498482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:03:47.032561665 (568h34m46.90364471s) [upload=498482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:03:47.349046430 (568h34m47.220129465s) [upload=498482]: Done uploading segments... Dec 29 08:03:47.349078150 (568h34m47.220161196s) [upload=498482]: Upload finished. err: <nil> Duration: 30.003030856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X..............X.X.X.............X............] duration: 34.711317441s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:03:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 34.711345965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................X........................X..X.] duration: 3.53355525s trace: https://jaeger.dev.storj.io/trace/361d49667c5f8ac6 FAIL (2025-12-29 08:03:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533577132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X.XX..X...X..XX..X...X.XXXX.X........X...X............X] duration: 30.002505117s trace: https://jaeger.dev.storj.io/trace/700c00e67b94eea9 FAIL (2025-12-29 08:03:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: TIY7C6YMNTKTDGPQ2ARZD466I7VGIFU544G6RX52XLUCIVAICLOA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: X5VSG2KZRHOKJ2PK64HK4445E7GCGJAEQPE2IDCMRF7S3NWHQZ7Q) context deadline exceeded ======LOGS====== ec 29 08:02:38.587257895 (568h33m38.45834094s) [upload=498465]: Starting upload Dec 29 08:02:38.587332006 (568h33m38.458415041s) [upload=498465]: Done starting upload Dec 29 08:02:38.587547668 (568h33m38.458630703s) [upload=498465]: Uploading segments... Dec 29 08:02:38.587629403 (568h33m38.458712448s) [upload=498465]: Got next segment. Inline: false Dec 29 08:02:38.587672835 (568h33m38.458755871s) [upload=498465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:02:38.587679458 (568h33m38.458762493s) [upload=498465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:02:38.883224651 (568h33m38.754307696s) [seg_pos={0 0}, upload=498465]: Begin upload segment... Dec 29 08:02:38.883272703 (568h33m38.754355738s) [seg_pos={0 0}, upload=498465]: Done begin upload segment. Dec 29 08:02:38.883304263 (568h33m38.754387298s) [noise=true , address=145.79.227.11:4732 , piece_id=X6WPUB5JDC23I3P7ZIPCAG5HDGJA6RN62ZUFN4APNLRGEFLUHC4A, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.883360830 (568h33m38.754443865s) [noise=true , address=77.90.31.10:4520 , piece_id=TIY7C6YMNTKTDGPQ2ARZD466I7VGIFU544G6RX52XLUCIVAICLOA, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.883846225 (568h33m38.75492926s) [noise=true , address=145.79.227.48:4302 , piece_id=GS2KDYDVJL5E7CIW5KGKFKSNWTMAKDT5LY5QU43G7PVPQ3FGX2LA, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.884190701 (568h33m38.755273736s) [noise=true , address=145.79.227.42:4122 , piece_id=GLMEMQ3AFYUXKRIKVS3WESQFRBFJMDR2BS2QTIUDHD6HL27FIJIA, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.884474982 (568h33m38.755558027s) [noise=true , address=145.79.227.49:4338 , piece_id=FLPHRLSF2ICL5ET7X525HJKPBFNVVMLFUQRJFIRCHAHWBGIGKEWQ, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.884844005 (568h33m38.75592704s) [noise=true , address=77.90.31.15:4697 , piece_id=5VLC55LVTXP6VFRVFJWJR6A6PT23UE3KFAA4VDDRGI2NHJKKO4BA, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.885228217 (568h33m38.756311262s) [noise=true , address=145.79.227.51:4410 , piece_id=U2G7LYCJ5QYFXFFGSWHXM3UQ5CNSW7QGDF6TR6OBVLYQZEDQBTUQ, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.885683984 (568h33m38.756767029s) [noise=true , address=145.79.227.6:4258 , piece_id=3IZUITJ7JZG5V2LEHYMOPMYUWUIQEBRSJBFIHMF7A2EXJEBW64GQ, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.886263667 (568h33m38.757346713s) [noise=true , address=45.91.116.11:4119 , piece_id=NYOOEPOMC3TTSQD6X3N5OLPW4ZABP7UREPWDIPSIU6DYQ22VT2TA, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.887527284 (568h33m38.758610329s) [noise=true , address=145.79.227.50:4373 , piece_id=ZQYRDJ2J7UOGFMWBCGFGCE3AZWZOGF5O6YXTVYLLET4Z6AM3CGTQ, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.887764476 (568h33m38.758847521s) [noise=true , address=145.79.227.16:4909 , piece_id=RLTBCF2EWNU7R3CIWN42CIX4UARSASYUTGLO7KKOFNIAJP73OL6Q, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.889036909 (568h33m38.760119944s) [noise=true , address=145.79.227.13:4803 , piece_id=62S6I7KL3UAYA3L2OLOYLVGCRNFA5R2T3MCFEJ3BVFUEAHOUWKYQ, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.889240787 (568h33m38.760323822s) [noise=true , address=145.79.227.29:4942 , piece_id=NYRWDYM3QL42EX4OOJ4TWV4APIBZGWZTDMLXM5DDRA2E4IO6XCDA, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.890587241 (568h33m38.761670276s) [noise=true , address=145.79.227.19:4077 , piece_id=ZJAWYSGD254VDPQ7ZAMJMFXHNHWQAXHYMDVCBJLHK3VGD7GN3O2Q, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.890831276 (568h33m38.761914311s) [noise=true , address=145.79.227.52:4445 , piece_id=PHYRRDOUG6VHKU5YNSH73S62P3XGWVCJZWDX6KR53CVGU2GCLAKA, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:38.892487779 (568h33m38.763570814s) [noise=true , address=45.91.116.10:4081 , piece_id=X5VSG2KZRHOKJ2PK64HK4445E7GCGJAEQPE2IDCMRF7S3NWHQZ7Q, seg_pos={0 0}, upload=498465]: Uploading piece... Dec 29 08:02:40.579256656 (568h33m40.450339691s) [upload=498465]: Next returned nil segment Dec 29 08:02:40.579270282 (568h33m40.450353327s) [upload=498465]: Waiting for error group managing segments... Dec 29 08:02:40.862643969 (568h33m40.733727004s) [noise=true , address=145.79.227.42:4122 , piece_id=GLMEMQ3AFYUXKRIKVS3WESQFRBFJMDR2BS2QTIUDHD6HL27FIJIA, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:40.871364033 (568h33m40.742447078s) [noise=true , address=145.79.227.48:4302 , piece_id=GS2KDYDVJL5E7CIW5KGKFKSNWTMAKDT5LY5QU43G7PVPQ3FGX2LA, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:40.881671941 (568h33m40.752754986s) [noise=true , address=145.79.227.13:4803 , piece_id=62S6I7KL3UAYA3L2OLOYLVGCRNFA5R2T3MCFEJ3BVFUEAHOUWKYQ, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:40.885946741 (568h33m40.757029786s) [noise=true , address=145.79.227.19:4077 , piece_id=ZJAWYSGD254VDPQ7ZAMJMFXHNHWQAXHYMDVCBJLHK3VGD7GN3O2Q, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:40.886233637 (568h33m40.757316672s) [noise=true , address=145.79.227.50:4373 , piece_id=ZQYRDJ2J7UOGFMWBCGFGCE3AZWZOGF5O6YXTVYLLET4Z6AM3CGTQ, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:40.894901632 (568h33m40.765984667s) [noise=true , address=145.79.227.6:4258 , piece_id=3IZUITJ7JZG5V2LEHYMOPMYUWUIQEBRSJBFIHMF7A2EXJEBW64GQ, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:40.898185916 (568h33m40.769268951s) [noise=true , address=145.79.227.51:4410 , piece_id=U2G7LYCJ5QYFXFFGSWHXM3UQ5CNSW7QGDF6TR6OBVLYQZEDQBTUQ, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:40.902614368 (568h33m40.773697413s) [noise=true , address=145.79.227.52:4445 , piece_id=PHYRRDOUG6VHKU5YNSH73S62P3XGWVCJZWDX6KR53CVGU2GCLAKA, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:40.909239544 (568h33m40.780322579s) [noise=true , address=145.79.227.11:4732 , piece_id=X6WPUB5JDC23I3P7ZIPCAG5HDGJA6RN62ZUFN4APNLRGEFLUHC4A, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:40.909520930 (568h33m40.780603965s) [noise=true , address=145.79.227.16:4909 , piece_id=RLTBCF2EWNU7R3CIWN42CIX4UARSASYUTGLO7KKOFNIAJP73OL6Q, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:41.268544094 (568h33m41.139627129s) [noise=true , address=45.91.116.11:4119 , piece_id=NYOOEPOMC3TTSQD6X3N5OLPW4ZABP7UREPWDIPSIU6DYQ22VT2TA, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:41.472641406 (568h33m41.343724451s) [noise=true , address=77.90.31.10:4520 , piece_id=TIY7C6YMNTKTDGPQ2ARZD466I7VGIFU544G6RX52XLUCIVAICLOA, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:41.516151255 (568h33m41.38723429s) [noise=true , address=45.91.116.10:4081 , piece_id=X5VSG2KZRHOKJ2PK64HK4445E7GCGJAEQPE2IDCMRF7S3NWHQZ7Q, seg_pos={0 0}, upload=498465]: Done uploading piece. err: <nil> Dec 29 08:02:41.516170172 (568h33m41.387253207s) [seg_pos={0 0}, upload=498465]: Segment reached optimal threshold of 13 pieces. Dec 29 08:02:41.516409878 (568h33m41.387492923s) [noise=true , address=145.79.227.29:4942 , piece_id=NYRWDYM3QL42EX4OOJ4TWV4APIBZGWZTDMLXM5DDRA2E4IO6XCDA, seg_pos={0 0}, upload=498465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 08:02:41.516422422 (568h33m41.387505457s) [noise=true , address=145.79.227.29:4942 , piece_id=NYRWDYM3QL42EX4OOJ4TWV4APIBZGWZTDMLXM5DDRA2E4IO6XCDA, seg_pos={0 0}, upload=498465]: Long tail cancellation due to optimal threshold Dec 29 08:02:41.516500370 (568h33m41.387583415s) [noise=true , address=145.79.227.49:4338 , piece_id=FLPHRLSF2ICL5ET7X525HJKPBFNVVMLFUQRJFIRCHAHWBGIGKEWQ, seg_pos={0 0}, upload=498465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 08:02:41.516508525 (568h33m41.38759156s) [noise=true , address=145.79.227.49:4338 , piece_id=FLPHRLSF2ICL5ET7X525HJKPBFNVVMLFUQRJFIRCHAHWBGIGKEWQ, seg_pos={0 0}, upload=498465]: Long tail cancellation due to optimal threshold Dec 29 08:02:41.516622934 (568h33m41.387705969s) [noise=true , address=77.90.31.15:4697 , piece_id=5VLC55LVTXP6VFRVFJWJR6A6PT23UE3KFAA4VDDRGI2NHJKKO4BA, seg_pos={0 0}, upload=498465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:56224->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:56224->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56224->77.90.31.15:4697: use of closed network connection Dec 29 08:02:41.516629917 (568h33m41.387712952s) [noise=true , address=77.90.31.15:4697 , piece_id=5VLC55LVTXP6VFRVFJWJR6A6PT23UE3KFAA4VDDRGI2NHJKKO4BA, seg_pos={0 0}, upload=498465]: Long tail cancellation due to optimal threshold Dec 29 08:02:41.516647590 (568h33m41.387730635s) [seg_pos={0 0}, upload=498465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:02:41.516742271 (568h33m41.387825316s) [upload=498465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:02:41.516749214 (568h33m41.387832259s) [upload=498465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:02:41.882005275 (568h33m41.75308832s) [upload=498465]: Done uploading segments... Dec 29 08:02:41.882028990 (568h33m41.753112025s) [upload=498465]: Upload finished. err: <nil> Duration: 30.002538713s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................X........................X..X] duration: 5.001591703s trace: https://jaeger.dev.storj.io/trace/361d49667c5f8ac6 FAIL (2025-12-29 08:02:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001614969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X..............X.X.X.............X...........] duration: 37.775961057s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:02:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 37.775982448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X.XX..X...X..XX..X...X.XXXX.X........X...X............] duration: 8.419432174s trace: https://jaeger.dev.storj.io/trace/3f33be5421f0021b FAIL (2025-12-29 08:02:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: MMRAUOFHLL3EGRL5QEWY2RRR4IA253SF2ZUP6EORWERI2DM7JM3A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: PYRNKBGUGHQIRZCVRMLRIM322CETHJURCBPXN77L5YFOX3LZ4EYQ) context deadline exceeded ======LOGS====== Dec 29 07:52:51.080903808 (568h23m50.951986843s) [upload=498318]: Starting upload Dec 29 07:52:51.080962348 (568h23m50.952045383s) [upload=498318]: Done starting upload Dec 29 07:52:51.081128976 (568h23m50.952212011s) [upload=498318]: Uploading segments... Dec 29 07:52:51.081250969 (568h23m50.952334004s) [upload=498318]: Got next segment. Inline: false Dec 29 07:52:51.081302095 (568h23m50.952385131s) [upload=498318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:52:51.081307626 (568h23m50.952390661s) [upload=498318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:52:51.325379808 (568h23m51.196462843s) [seg_pos={0 0}, upload=498318]: Begin upload segment... Dec 29 07:52:51.325425956 (568h23m51.196508991s) [seg_pos={0 0}, upload=498318]: Done begin upload segment. Dec 29 07:52:51.325458327 (568h23m51.196541362s) [noise=true , address=77.90.31.13:4627 , piece_id=RDTDLQJBJF7YJRE7WMDL7NYDFVRBGWASFB3WXOCLETQ2KQVJB6VA, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.325472715 (568h23m51.19655575s) [noise=true , address=77.90.31.10:4519 , piece_id=YY3A6HEUV5XMUQWWUVTGCZNQAX3GYR3IFTAX3N4R2J4OUVDNOSAQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.326149494 (568h23m51.197232529s) [noise=true , address=145.79.227.47:4266 , piece_id=MZTZWX4BCKSAKO2NJNRVJRA6OLOO2OQRYJPOI6ERMHMMEYAKMOHA, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.326415250 (568h23m51.197498285s) [noise=true , address=145.79.227.46:4601 , piece_id=I6MQM7LYMICEEEW47TBCOWPLFSWEJOTWVMHDN5OEQTJGYJ4DAVJA, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.326700904 (568h23m51.197783939s) [noise=true , address=145.79.227.45:4231 , piece_id=OWO7QBKKUEOYZOLX76VMGIETK7Q4CEESAE6DGOY5ZZUUHSTPT5CQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.327012186 (568h23m51.198095232s) [noise=true , address=145.79.227.33:4870 , piece_id=QA5AYUTKOTDH7XSARJZCTK7FKC7KB6364UR5SAIZ7DGIA543S4VQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.327513471 (568h23m51.198596506s) [noise=true , address=145.79.227.7:4295 , piece_id=IDUPF72TH3WFW656PMPFTH6XAUPWOAUDZWTDDGKETFGNO6SKXXOQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.328063949 (568h23m51.199146984s) [noise=true , address=77.90.31.18:4808 , piece_id=2GGRQUOUNHU4HWXN2QEHBJV3BUYNWUU64DQ4TEP4TPSHBFS72I5A, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.328085340 (568h23m51.199168375s) [noise=true , address=77.90.31.14:4661 , piece_id=MMRAUOFHLL3EGRL5QEWY2RRR4IA253SF2ZUP6EORWERI2DM7JM3A, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.328452238 (568h23m51.199535273s) [noise=true , address=145.79.227.36:4517 , piece_id=7WPBKV2UMQTAUFABP2LDZXVEQNT4T36L4MZ4RW5AJZPKMHKFU24A, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.328579411 (568h23m51.199662456s) [noise=true , address=77.90.31.12:4589 , piece_id=PYRNKBGUGHQIRZCVRMLRIM322CETHJURCBPXN77L5YFOX3LZ4EYQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.328785714 (568h23m51.199868749s) [noise=true , address=145.79.227.48:4301 , piece_id=W4ZOJJOVTMS2NEPS6CUEJ6R7IPJKGALQPMK7OJWIQ2644GSCHS6A, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.329124599 (568h23m51.200207644s) [noise=true , address=145.79.227.13:4802 , piece_id=TMMHVXPXKXUF7OYTQUNJSHRE7G2PKICRYMLU43V4LNCMDCCWV3SQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.329393230 (568h23m51.200476275s) [noise=true , address=145.79.227.38:4978 , piece_id=XDWLV7KGFMFSQHX6YJNP3YBXEEAG7DVCMRWKIHDRUEE64V5A36DA, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.329662122 (568h23m51.200745157s) [noise=true , address=145.79.227.17:4970 , piece_id=YHEX2X7S5BWBKT6Y4JQSDZ2EK2GHLA6HDW4YPF7TLFZRWRJ56PDQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.329913891 (568h23m51.200996926s) [noise=true , address=145.79.227.26:4654 , piece_id=NS67L7ESJDRWI2Q4D7SRPRWVCXLT6PQUQJWYO3Y7YJUZLJSO36BA, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:53.025993688 (568h23m52.897076723s) [upload=498318]: Next returned nil segment Dec 29 07:52:53.026039696 (568h23m52.897122761s) [upload=498318]: Waiting for error group managing segments... Dec 29 07:52:53.309326076 (568h23m53.180409111s) [noise=true , address=145.79.227.46:4601 , piece_id=I6MQM7LYMICEEEW47TBCOWPLFSWEJOTWVMHDN5OEQTJGYJ4DAVJA, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.315468371 (568h23m53.186551407s) [noise=true , address=145.79.227.38:4978 , piece_id=XDWLV7KGFMFSQHX6YJNP3YBXEEAG7DVCMRWKIHDRUEE64V5A36DA, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.317036888 (568h23m53.188119923s) [noise=true , address=145.79.227.48:4301 , piece_id=W4ZOJJOVTMS2NEPS6CUEJ6R7IPJKGALQPMK7OJWIQ2644GSCHS6A, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.317470685 (568h23m53.18855373s) [noise=true , address=145.79.227.26:4654 , piece_id=NS67L7ESJDRWI2Q4D7SRPRWVCXLT6PQUQJWYO3Y7YJUZLJSO36BA, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.319014785 (568h23m53.19009782s) [noise=true , address=145.79.227.36:4517 , piece_id=7WPBKV2UMQTAUFABP2LDZXVEQNT4T36L4MZ4RW5AJZPKMHKFU24A, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.327000741 (568h23m53.198083776s) [noise=true , address=145.79.227.13:4802 , piece_id=TMMHVXPXKXUF7OYTQUNJSHRE7G2PKICRYMLU43V4LNCMDCCWV3SQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.330738027 (568h23m53.201821072s) [noise=true , address=145.79.227.45:4231 , piece_id=OWO7QBKKUEOYZOLX76VMGIETK7Q4CEESAE6DGOY5ZZUUHSTPT5CQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.333404084 (568h23m53.204487119s) [noise=true , address=145.79.227.7:4295 , piece_id=IDUPF72TH3WFW656PMPFTH6XAUPWOAUDZWTDDGKETFGNO6SKXXOQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.342076197 (568h23m53.213159232s) [noise=true , address=145.79.227.17:4970 , piece_id=YHEX2X7S5BWBKT6Y4JQSDZ2EK2GHLA6HDW4YPF7TLFZRWRJ56PDQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.348279659 (568h23m53.219362704s) [noise=true , address=145.79.227.47:4266 , piece_id=MZTZWX4BCKSAKO2NJNRVJRA6OLOO2OQRYJPOI6ERMHMMEYAKMOHA, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.538296070 (568h23m53.409379105s) [noise=true , address=77.90.31.12:4589 , piece_id=PYRNKBGUGHQIRZCVRMLRIM322CETHJURCBPXN77L5YFOX3LZ4EYQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.578853998 (568h23m53.449937033s) [noise=true , address=77.90.31.13:4627 , piece_id=RDTDLQJBJF7YJRE7WMDL7NYDFVRBGWASFB3WXOCLETQ2KQVJB6VA, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.610813784 (568h23m53.481896819s) [noise=true , address=77.90.31.14:4661 , piece_id=MMRAUOFHLL3EGRL5QEWY2RRR4IA253SF2ZUP6EORWERI2DM7JM3A, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.610831027 (568h23m53.481914062s) [seg_pos={0 0}, upload=498318]: Segment reached optimal threshold of 13 pieces. Dec 29 07:52:53.611002193 (568h23m53.482085228s) [noise=true , address=145.79.227.33:4870 , piece_id=QA5AYUTKOTDH7XSARJZCTK7FKC7KB6364UR5SAIZ7DGIA543S4VQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:53.611008164 (568h23m53.482091199s) [noise=true , address=145.79.227.33:4870 , piece_id=QA5AYUTKOTDH7XSARJZCTK7FKC7KB6364UR5SAIZ7DGIA543S4VQ, seg_pos={0 0}, upload=498318]: Long tail cancellation due to optimal threshold Dec 29 07:52:53.611179741 (568h23m53.482262776s) [noise=true , address=77.90.31.10:4519 , piece_id=YY3A6HEUV5XMUQWWUVTGCZNQAX3GYR3IFTAX3N4R2J4OUVDNOSAQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:53.611187035 (568h23m53.48227007s) [noise=true , address=77.90.31.10:4519 , piece_id=YY3A6HEUV5XMUQWWUVTGCZNQAX3GYR3IFTAX3N4R2J4OUVDNOSAQ, seg_pos={0 0}, upload=498318]: Long tail cancellation due to optimal threshold Dec 29 07:52:53.611241790 (568h23m53.482324835s) [noise=true , address=77.90.31.18:4808 , piece_id=2GGRQUOUNHU4HWXN2QEHBJV3BUYNWUU64DQ4TEP4TPSHBFS72I5A, seg_pos={0 0}, upload=498318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:53.611254042 (568h23m53.482337098s) [noise=true , address=77.90.31.18:4808 , piece_id=2GGRQUOUNHU4HWXN2QEHBJV3BUYNWUU64DQ4TEP4TPSHBFS72I5A, seg_pos={0 0}, upload=498318]: Long tail cancellation due to optimal threshold Dec 29 07:52:53.611309998 (568h23m53.482393044s) [seg_pos={0 0}, upload=498318]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:52:53.611522274 (568h23m53.482605329s) [upload=498318]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:53.611530178 (568h23m53.482613213s) [upload=498318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:53.895812495 (568h23m53.766895541s) [upload=498318]: Done uploading segments... Dec 29 07:52:53.895836280 (568h23m53.766919335s) [upload=498318]: Upload finished. err: <nil> Duration: 8.419457061s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X........................X..] duration: 3.594189668s trace: https://jaeger.dev.storj.io/trace/79d6185f6dead8ad FAIL (2025-12-29 08:01:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.594219715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X..............X.X.X.............X..........] duration: 39.064362768s trace: https://jaeger.dev.storj.io/trace/3ed58f4215e12d2d FAIL (2025-12-29 08:01:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== HMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816124105 (568h23m36.68720714s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816466777 (568h23m36.687549813s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.816776638 (568h23m36.687859683s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817049397 (568h23m36.688132432s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817370158 (568h23m36.688453193s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.817768567 (568h23m36.688851602s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818348320 (568h23m36.689431355s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.818777107 (568h23m36.689860152s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819188750 (568h23m36.690271795s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.819629219 (568h23m36.690712255s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.820663369 (568h23m36.691746404s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.822280047 (568h23m36.693363082s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.823054541 (568h23m36.694137577s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.829118048 (568h23m36.700201083s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.831625562 (568h23m36.702708607s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:36.834289415 (568h23m36.70537245s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Uploading piece... Dec 29 07:52:37.753701044 (568h23m37.624784079s) [noise=true , address=64.49.13.27:4966 , piece_id=OIJTDMAT52ZUDLJJVWFOUVBPBAKF5VU3XJBUFN63OJH36TVEGCFA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: ecclient: upload failed (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5, address:64.49.13.27:4966): protocol: storage node overloaded, request limit: 2000 Dec 29 07:52:38.183880030 (568h23m38.054963075s) [upload=498312]: Next returned nil segment Dec 29 07:52:38.183895088 (568h23m38.054978133s) [upload=498312]: Waiting for error group managing segments... Dec 29 07:52:38.564630267 (568h23m38.435713302s) [noise=true , address=64.49.13.170:4720 , piece_id=E7SSKER6I7XNOFL2FW4FZBF6DPILOK3W7Y33X3SBFHRFAQEUWHFQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.669750161 (568h23m38.540833196s) [noise=true , address=145.79.227.52:4459 , piece_id=A5JFNMWNZT6FEWPUM76RM5J6J4TKGNRC5OVNS5VF2QXR5UOB644A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.687406973 (568h23m38.558490008s) [noise=true , address=145.79.227.12:4776 , piece_id=ONDBZJ4ZD7F3DT4LQVFIIL4PHDRGTCSMXDI63VXUHLWBDICREH3Q, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.693860731 (568h23m38.564943766s) [noise=true , address=145.79.227.3:4155 , piece_id=BQGSBHAEW6WJCTVONOW75UFL2ZCKDYJ2L64JYYDJ53EDQTCX3LGQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.878909343 (568h23m38.749992388s) [noise=true , address=77.90.31.21:4906 , piece_id=3MCFNKJNAKGRF6BYPB7TTJM233DNASO2IUNJBT4SF22EVGTX22GA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.949239613 (568h23m38.820322648s) [noise=true , address=45.91.116.12:4239 , piece_id=ZPPIXLPLOF7ARX6K6KO63SC747VVMD2JSNJ7LLXCEXMUH53553TA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:38.980416277 (568h23m38.851499312s) [noise=true , address=77.90.31.17:4785 , piece_id=PBVWGOAEXTSXHMHIET3LQ3OH56FCDXZ5SSIFSHRHV7GHFQTUVL7A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.015124567 (568h23m38.886207612s) [noise=true , address=77.90.31.18:4818 , piece_id=KTQI2KMJOYEKGCIAXUJ2DNJSNSR4P5QKTCESNRYRNPXBZN3EDXHA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.518277718 (568h23m39.389360753s) [noise=true , address=145.79.227.44:4219 , piece_id=W6UEWJCVWOZ2TUB5EVUIUK3NDOJS4BYX7ZAEOME75TLBSZ3HKCPA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555849801 (568h23m39.426932846s) [noise=true , address=193.149.164.143:4704 , piece_id=HZ5CSH2J6BTLRLR2UGQ6T7J2OTW356M2G7EW6CCRONSQISAJTURQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.555935634 (568h23m39.427018679s) [noise=true , address=145.79.227.35:4496 , piece_id=OMXDWGS6KQE7XR2C7WXTXTDCC6STOJOR6ZTSBGU4KDRSACR3SZRA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.558751436 (568h23m39.429834471s) [noise=true , address=145.79.227.43:4160 , piece_id=2C7N4CWLJUXMEHJELSQSYJ53FAJJHNEK6LENGV2O4ZBTZGMMZD6A, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.595945009 (568h23m39.467028044s) [noise=true , address=145.79.227.30:4767 , piece_id=JPDITEN7UBRPXSMJWZ56MRZ2LFVJJT73YGZIXUM7VHX45BTPRTVA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611829505 (568h23m39.48291254s) [noise=true , address=145.79.227.33:4888 , piece_id=CP35D54N4T5SLVDGQ6RUJMSWBISVAEEOJ37GGHUT5ZVRAIGHS7HA, seg_pos={0 0}, upload=498312]: Done uploading piece. err: <nil> Dec 29 07:52:39.611854823 (568h23m39.482937858s) [seg_pos={0 0}, upload=498312]: Segment reached optimal threshold of 14 pieces. Dec 29 07:52:39.612134715 (568h23m39.483217761s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:39.612143904 (568h23m39.483226949s) [noise=true , address=145.79.227.34:4924 , piece_id=VHXAZ74FQY7G6DW2HEHY6XJM4UUGGAYIIEZ7AD7GADETQQAZROLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612514058 (568h23m39.483597113s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48592->64.49.13.169:4709: use of closed network connection Dec 29 07:52:39.612533576 (568h23m39.483616631s) [noise=true , address=64.49.13.169:4709 , piece_id=2NVT3BFD6SKIDEOUZFERGJGZDODLZQA7FW7QFONBPYOIQCZLJUMQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612556999 (568h23m39.483640035s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18tmori7tiA7FvTDd5MXLX7F2DoAxg3QFNtZHwCMnVtgwP7uve): protocol: read tcp 188.166.225.170:35550->45.91.116.4:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:52:39.612564755 (568h23m39.48364779s) [noise=true , address=45.91.116.4:4946 , piece_id=OHZZGNFAK3TYX5JA4SVTIV7V6D5ZRZFTTDGWCWQBN4YJKRCRJDLQ, seg_pos={0 0}, upload=498312]: Long tail cancellation due to optimal threshold Dec 29 07:52:39.612679874 (568h23m39.483762919s) [seg_pos={0 0}, upload=498312]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:52:39.612755728 (568h23m39.483838763s) [upload=498312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:39.612760287 (568h23m39.483843322s) [upload=498312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:39.902784558 (568h23m39.773867603s) [upload=498312]: Done uploading segments... Dec 29 07:52:39.902807752 (568h23m39.773890797s) [upload=498312]: Upload finished. err: <nil> Duration: 39.064382525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......X.XX..X...X..XX..X...X.XXXX.X........X...X...........] duration: 6.066041823s trace: https://jaeger.dev.storj.io/trace/3f33be5421f0021b FAIL (2025-12-29 08:01:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: MMRAUOFHLL3EGRL5QEWY2RRR4IA253SF2ZUP6EORWERI2DM7JM3A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: PYRNKBGUGHQIRZCVRMLRIM322CETHJURCBPXN77L5YFOX3LZ4EYQ) context deadline exceeded ======LOGS====== Dec 29 07:52:51.080903808 (568h23m50.951986843s) [upload=498318]: Starting upload Dec 29 07:52:51.080962348 (568h23m50.952045383s) [upload=498318]: Done starting upload Dec 29 07:52:51.081128976 (568h23m50.952212011s) [upload=498318]: Uploading segments... Dec 29 07:52:51.081250969 (568h23m50.952334004s) [upload=498318]: Got next segment. Inline: false Dec 29 07:52:51.081302095 (568h23m50.952385131s) [upload=498318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:52:51.081307626 (568h23m50.952390661s) [upload=498318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:52:51.325379808 (568h23m51.196462843s) [seg_pos={0 0}, upload=498318]: Begin upload segment... Dec 29 07:52:51.325425956 (568h23m51.196508991s) [seg_pos={0 0}, upload=498318]: Done begin upload segment. Dec 29 07:52:51.325458327 (568h23m51.196541362s) [noise=true , address=77.90.31.13:4627 , piece_id=RDTDLQJBJF7YJRE7WMDL7NYDFVRBGWASFB3WXOCLETQ2KQVJB6VA, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.325472715 (568h23m51.19655575s) [noise=true , address=77.90.31.10:4519 , piece_id=YY3A6HEUV5XMUQWWUVTGCZNQAX3GYR3IFTAX3N4R2J4OUVDNOSAQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.326149494 (568h23m51.197232529s) [noise=true , address=145.79.227.47:4266 , piece_id=MZTZWX4BCKSAKO2NJNRVJRA6OLOO2OQRYJPOI6ERMHMMEYAKMOHA, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.326415250 (568h23m51.197498285s) [noise=true , address=145.79.227.46:4601 , piece_id=I6MQM7LYMICEEEW47TBCOWPLFSWEJOTWVMHDN5OEQTJGYJ4DAVJA, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.326700904 (568h23m51.197783939s) [noise=true , address=145.79.227.45:4231 , piece_id=OWO7QBKKUEOYZOLX76VMGIETK7Q4CEESAE6DGOY5ZZUUHSTPT5CQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.327012186 (568h23m51.198095232s) [noise=true , address=145.79.227.33:4870 , piece_id=QA5AYUTKOTDH7XSARJZCTK7FKC7KB6364UR5SAIZ7DGIA543S4VQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.327513471 (568h23m51.198596506s) [noise=true , address=145.79.227.7:4295 , piece_id=IDUPF72TH3WFW656PMPFTH6XAUPWOAUDZWTDDGKETFGNO6SKXXOQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.328063949 (568h23m51.199146984s) [noise=true , address=77.90.31.18:4808 , piece_id=2GGRQUOUNHU4HWXN2QEHBJV3BUYNWUU64DQ4TEP4TPSHBFS72I5A, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.328085340 (568h23m51.199168375s) [noise=true , address=77.90.31.14:4661 , piece_id=MMRAUOFHLL3EGRL5QEWY2RRR4IA253SF2ZUP6EORWERI2DM7JM3A, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.328452238 (568h23m51.199535273s) [noise=true , address=145.79.227.36:4517 , piece_id=7WPBKV2UMQTAUFABP2LDZXVEQNT4T36L4MZ4RW5AJZPKMHKFU24A, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.328579411 (568h23m51.199662456s) [noise=true , address=77.90.31.12:4589 , piece_id=PYRNKBGUGHQIRZCVRMLRIM322CETHJURCBPXN77L5YFOX3LZ4EYQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.328785714 (568h23m51.199868749s) [noise=true , address=145.79.227.48:4301 , piece_id=W4ZOJJOVTMS2NEPS6CUEJ6R7IPJKGALQPMK7OJWIQ2644GSCHS6A, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.329124599 (568h23m51.200207644s) [noise=true , address=145.79.227.13:4802 , piece_id=TMMHVXPXKXUF7OYTQUNJSHRE7G2PKICRYMLU43V4LNCMDCCWV3SQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.329393230 (568h23m51.200476275s) [noise=true , address=145.79.227.38:4978 , piece_id=XDWLV7KGFMFSQHX6YJNP3YBXEEAG7DVCMRWKIHDRUEE64V5A36DA, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.329662122 (568h23m51.200745157s) [noise=true , address=145.79.227.17:4970 , piece_id=YHEX2X7S5BWBKT6Y4JQSDZ2EK2GHLA6HDW4YPF7TLFZRWRJ56PDQ, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:51.329913891 (568h23m51.200996926s) [noise=true , address=145.79.227.26:4654 , piece_id=NS67L7ESJDRWI2Q4D7SRPRWVCXLT6PQUQJWYO3Y7YJUZLJSO36BA, seg_pos={0 0}, upload=498318]: Uploading piece... Dec 29 07:52:53.025993688 (568h23m52.897076723s) [upload=498318]: Next returned nil segment Dec 29 07:52:53.026039696 (568h23m52.897122761s) [upload=498318]: Waiting for error group managing segments... Dec 29 07:52:53.309326076 (568h23m53.180409111s) [noise=true , address=145.79.227.46:4601 , piece_id=I6MQM7LYMICEEEW47TBCOWPLFSWEJOTWVMHDN5OEQTJGYJ4DAVJA, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.315468371 (568h23m53.186551407s) [noise=true , address=145.79.227.38:4978 , piece_id=XDWLV7KGFMFSQHX6YJNP3YBXEEAG7DVCMRWKIHDRUEE64V5A36DA, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.317036888 (568h23m53.188119923s) [noise=true , address=145.79.227.48:4301 , piece_id=W4ZOJJOVTMS2NEPS6CUEJ6R7IPJKGALQPMK7OJWIQ2644GSCHS6A, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.317470685 (568h23m53.18855373s) [noise=true , address=145.79.227.26:4654 , piece_id=NS67L7ESJDRWI2Q4D7SRPRWVCXLT6PQUQJWYO3Y7YJUZLJSO36BA, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.319014785 (568h23m53.19009782s) [noise=true , address=145.79.227.36:4517 , piece_id=7WPBKV2UMQTAUFABP2LDZXVEQNT4T36L4MZ4RW5AJZPKMHKFU24A, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.327000741 (568h23m53.198083776s) [noise=true , address=145.79.227.13:4802 , piece_id=TMMHVXPXKXUF7OYTQUNJSHRE7G2PKICRYMLU43V4LNCMDCCWV3SQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.330738027 (568h23m53.201821072s) [noise=true , address=145.79.227.45:4231 , piece_id=OWO7QBKKUEOYZOLX76VMGIETK7Q4CEESAE6DGOY5ZZUUHSTPT5CQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.333404084 (568h23m53.204487119s) [noise=true , address=145.79.227.7:4295 , piece_id=IDUPF72TH3WFW656PMPFTH6XAUPWOAUDZWTDDGKETFGNO6SKXXOQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.342076197 (568h23m53.213159232s) [noise=true , address=145.79.227.17:4970 , piece_id=YHEX2X7S5BWBKT6Y4JQSDZ2EK2GHLA6HDW4YPF7TLFZRWRJ56PDQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.348279659 (568h23m53.219362704s) [noise=true , address=145.79.227.47:4266 , piece_id=MZTZWX4BCKSAKO2NJNRVJRA6OLOO2OQRYJPOI6ERMHMMEYAKMOHA, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.538296070 (568h23m53.409379105s) [noise=true , address=77.90.31.12:4589 , piece_id=PYRNKBGUGHQIRZCVRMLRIM322CETHJURCBPXN77L5YFOX3LZ4EYQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.578853998 (568h23m53.449937033s) [noise=true , address=77.90.31.13:4627 , piece_id=RDTDLQJBJF7YJRE7WMDL7NYDFVRBGWASFB3WXOCLETQ2KQVJB6VA, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.610813784 (568h23m53.481896819s) [noise=true , address=77.90.31.14:4661 , piece_id=MMRAUOFHLL3EGRL5QEWY2RRR4IA253SF2ZUP6EORWERI2DM7JM3A, seg_pos={0 0}, upload=498318]: Done uploading piece. err: <nil> Dec 29 07:52:53.610831027 (568h23m53.481914062s) [seg_pos={0 0}, upload=498318]: Segment reached optimal threshold of 13 pieces. Dec 29 07:52:53.611002193 (568h23m53.482085228s) [noise=true , address=145.79.227.33:4870 , piece_id=QA5AYUTKOTDH7XSARJZCTK7FKC7KB6364UR5SAIZ7DGIA543S4VQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:53.611008164 (568h23m53.482091199s) [noise=true , address=145.79.227.33:4870 , piece_id=QA5AYUTKOTDH7XSARJZCTK7FKC7KB6364UR5SAIZ7DGIA543S4VQ, seg_pos={0 0}, upload=498318]: Long tail cancellation due to optimal threshold Dec 29 07:52:53.611179741 (568h23m53.482262776s) [noise=true , address=77.90.31.10:4519 , piece_id=YY3A6HEUV5XMUQWWUVTGCZNQAX3GYR3IFTAX3N4R2J4OUVDNOSAQ, seg_pos={0 0}, upload=498318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:53.611187035 (568h23m53.48227007s) [noise=true , address=77.90.31.10:4519 , piece_id=YY3A6HEUV5XMUQWWUVTGCZNQAX3GYR3IFTAX3N4R2J4OUVDNOSAQ, seg_pos={0 0}, upload=498318]: Long tail cancellation due to optimal threshold Dec 29 07:52:53.611241790 (568h23m53.482324835s) [noise=true , address=77.90.31.18:4808 , piece_id=2GGRQUOUNHU4HWXN2QEHBJV3BUYNWUU64DQ4TEP4TPSHBFS72I5A, seg_pos={0 0}, upload=498318]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 07:52:53.611254042 (568h23m53.482337098s) [noise=true , address=77.90.31.18:4808 , piece_id=2GGRQUOUNHU4HWXN2QEHBJV3BUYNWUU64DQ4TEP4TPSHBFS72I5A, seg_pos={0 0}, upload=498318]: Long tail cancellation due to optimal threshold Dec 29 07:52:53.611309998 (568h23m53.482393044s) [seg_pos={0 0}, upload=498318]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:52:53.611522274 (568h23m53.482605329s) [upload=498318]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:52:53.611530178 (568h23m53.482613213s) [upload=498318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:52:53.895812495 (568h23m53.766895541s) [upload=498318]: Done uploading segments... Dec 29 07:52:53.895836280 (568h23m53.766919335s) [upload=498318]: Upload finished. err: <nil> Duration: 6.066069335s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)