Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 333327 reports (Page 108 of 6667)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...........X.............X..................................]
duration: 
34.838339221s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 06:00:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 34.838358047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [XX....XXX...X..XX.X.XX...XX......XX.X.X......X.....X..X.XX..]
duration: 
5.967401491s
trace: 
https://jaeger.dev.storj.io/trace/6e7b1f1cd9efe572
FAIL (2025-12-27 06:00:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA) context deadline exceeded
======LOGS======
 05:58:54.027618010 (518h29m53.898701045s) [upload=454908]: Starting upload
Dec 27 05:58:54.027674568 (518h29m53.898757613s) [upload=454908]: Done starting upload
Dec 27 05:58:54.027752656 (518h29m53.898835691s) [upload=454908]: Uploading segments...
Dec 27 05:58:54.027867967 (518h29m53.898951002s) [upload=454908]: Got next segment. Inline: false
Dec 27 05:58:54.027959360 (518h29m53.899042395s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:58:54.027966052 (518h29m53.899049088s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:58:54.273197646 (518h29m54.144280681s) [seg_pos={0 0}, upload=454908]: Begin upload segment...
Dec 27 05:58:54.273253432 (518h29m54.144336467s) [seg_pos={0 0}, upload=454908]: Done begin upload segment.
Dec 27 05:58:54.273335729 (518h29m54.144418764s) [noise=true , address=145.79.227.6:4259    , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.273355857 (518h29m54.144438902s) [noise=true , address=145.79.227.31:4797   , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.274397771 (518h29m54.145480816s) [noise=true , address=45.91.116.7:4009     , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.275285933 (518h29m54.146368968s) [noise=true , address=145.79.227.38:4978   , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.275978290 (518h29m54.147061335s) [noise=true , address=77.90.31.11:4556     , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.277650785 (518h29m54.14873382s) [noise=true , address=77.90.31.21:4877     , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.281356081 (518h29m54.152439116s) [noise=true , address=145.79.227.45:4231   , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.283248223 (518h29m54.154331268s) [noise=true , address=145.79.227.39:4013   , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.286543088 (518h29m54.157626123s) [noise=true , address=45.91.116.10:4084    , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.296837890 (518h29m54.167920935s) [noise=true , address=145.79.227.17:4971   , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.297626512 (518h29m54.168709547s) [noise=true , address=145.79.227.28:4725   , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.298856263 (518h29m54.169939298s) [noise=true , address=77.90.31.19:4842     , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.299710430 (518h29m54.170793485s) [noise=true , address=77.90.31.14:4661     , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.300773284 (518h29m54.171856319s) [noise=true , address=45.91.116.12:4226    , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.301333490 (518h29m54.172416535s) [noise=true , address=145.79.227.50:4374   , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.302620932 (518h29m54.173703967s) [noise=true , address=145.79.227.46:4603   , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:55.915655523 (518h29m55.786738558s) [upload=454908]: Next returned nil segment
Dec 27 05:58:55.915665622 (518h29m55.786748657s) [upload=454908]: Waiting for error group managing segments...
Dec 27 05:58:56.302185873 (518h29m56.173268918s) [noise=true , address=145.79.227.28:4725   , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.305487398 (518h29m56.176570434s) [noise=true , address=145.79.227.45:4231   , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.305946472 (518h29m56.177029507s) [noise=true , address=145.79.227.17:4971   , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.310843968 (518h29m56.181927003s) [noise=true , address=145.79.227.50:4374   , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.313521086 (518h29m56.184604141s) [noise=true , address=145.79.227.38:4978   , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.317048212 (518h29m56.188131247s) [noise=true , address=145.79.227.39:4013   , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.317491997 (518h29m56.188575043s) [noise=true , address=145.79.227.31:4797   , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.327595456 (518h29m56.198678491s) [noise=true , address=145.79.227.6:4259    , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.332749460 (518h29m56.203832505s) [noise=true , address=145.79.227.46:4603   , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.467173567 (518h29m56.338256602s) [noise=true , address=77.90.31.14:4661     , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.555793674 (518h29m56.426876719s) [noise=true , address=45.91.116.7:4009     , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.612756451 (518h29m56.483839486s) [noise=true , address=77.90.31.11:4556     , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.663674767 (518h29m56.534757813s) [noise=true , address=45.91.116.10:4084    , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.663704895 (518h29m56.53478794s) [seg_pos={0 0}, upload=454908]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:58:56.663995689 (518h29m56.535078754s) [noise=true , address=77.90.31.19:4842     , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection
Dec 27 05:58:56.664001861 (518h29m56.535084906s) [noise=true , address=77.90.31.19:4842     , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold
Dec 27 05:58:56.664126127 (518h29m56.535209172s) [noise=true , address=45.91.116.12:4226    , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 27 05:58:56.664169029 (518h29m56.535252065s) [noise=true , address=45.91.116.12:4226    , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold
Dec 27 05:58:56.664391382 (518h29m56.535474418s) [noise=true , address=77.90.31.21:4877     , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 27 05:58:56.664398546 (518h29m56.535481591s) [noise=true , address=77.90.31.21:4877     , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold
Dec 27 05:58:56.664419476 (518h29m56.535502521s) [seg_pos={0 0}, upload=454908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:58:56.664503536 (518h29m56.535586571s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:58:56.664509056 (518h29m56.535592091s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:58:56.958441263 (518h29m56.829524308s) [upload=454908]: Done uploading segments...
Dec 27 05:58:56.958480828 (518h29m56.829563863s) [upload=454908]: Upload finished. err: <nil>
Duration: 5.967418553s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.........................................X..................]
duration: 
3.628822611s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 06:00:39 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.628848882s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [............X.............X.................................]
duration: 
45.855573445s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 06:00:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 45.855596999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..........................................X.................]
duration: 
3.462782822s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 06:00:18 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.462825302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [XXX....XXX...X..XX.X.XX...XX......XX.X.X......X.....X..X.XX.]
duration: 
6.130854835s
trace: 
https://jaeger.dev.storj.io/trace/6e7b1f1cd9efe572
FAIL (2025-12-27 06:00:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA) context deadline exceeded
======LOGS======
 05:58:54.027618010 (518h29m53.898701045s) [upload=454908]: Starting upload
Dec 27 05:58:54.027674568 (518h29m53.898757613s) [upload=454908]: Done starting upload
Dec 27 05:58:54.027752656 (518h29m53.898835691s) [upload=454908]: Uploading segments...
Dec 27 05:58:54.027867967 (518h29m53.898951002s) [upload=454908]: Got next segment. Inline: false
Dec 27 05:58:54.027959360 (518h29m53.899042395s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:58:54.027966052 (518h29m53.899049088s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:58:54.273197646 (518h29m54.144280681s) [seg_pos={0 0}, upload=454908]: Begin upload segment...
Dec 27 05:58:54.273253432 (518h29m54.144336467s) [seg_pos={0 0}, upload=454908]: Done begin upload segment.
Dec 27 05:58:54.273335729 (518h29m54.144418764s) [noise=true , address=145.79.227.6:4259    , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.273355857 (518h29m54.144438902s) [noise=true , address=145.79.227.31:4797   , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.274397771 (518h29m54.145480816s) [noise=true , address=45.91.116.7:4009     , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.275285933 (518h29m54.146368968s) [noise=true , address=145.79.227.38:4978   , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.275978290 (518h29m54.147061335s) [noise=true , address=77.90.31.11:4556     , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.277650785 (518h29m54.14873382s) [noise=true , address=77.90.31.21:4877     , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.281356081 (518h29m54.152439116s) [noise=true , address=145.79.227.45:4231   , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.283248223 (518h29m54.154331268s) [noise=true , address=145.79.227.39:4013   , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.286543088 (518h29m54.157626123s) [noise=true , address=45.91.116.10:4084    , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.296837890 (518h29m54.167920935s) [noise=true , address=145.79.227.17:4971   , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.297626512 (518h29m54.168709547s) [noise=true , address=145.79.227.28:4725   , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.298856263 (518h29m54.169939298s) [noise=true , address=77.90.31.19:4842     , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.299710430 (518h29m54.170793485s) [noise=true , address=77.90.31.14:4661     , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.300773284 (518h29m54.171856319s) [noise=true , address=45.91.116.12:4226    , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.301333490 (518h29m54.172416535s) [noise=true , address=145.79.227.50:4374   , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.302620932 (518h29m54.173703967s) [noise=true , address=145.79.227.46:4603   , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:55.915655523 (518h29m55.786738558s) [upload=454908]: Next returned nil segment
Dec 27 05:58:55.915665622 (518h29m55.786748657s) [upload=454908]: Waiting for error group managing segments...
Dec 27 05:58:56.302185873 (518h29m56.173268918s) [noise=true , address=145.79.227.28:4725   , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.305487398 (518h29m56.176570434s) [noise=true , address=145.79.227.45:4231   , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.305946472 (518h29m56.177029507s) [noise=true , address=145.79.227.17:4971   , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.310843968 (518h29m56.181927003s) [noise=true , address=145.79.227.50:4374   , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.313521086 (518h29m56.184604141s) [noise=true , address=145.79.227.38:4978   , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.317048212 (518h29m56.188131247s) [noise=true , address=145.79.227.39:4013   , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.317491997 (518h29m56.188575043s) [noise=true , address=145.79.227.31:4797   , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.327595456 (518h29m56.198678491s) [noise=true , address=145.79.227.6:4259    , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.332749460 (518h29m56.203832505s) [noise=true , address=145.79.227.46:4603   , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.467173567 (518h29m56.338256602s) [noise=true , address=77.90.31.14:4661     , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.555793674 (518h29m56.426876719s) [noise=true , address=45.91.116.7:4009     , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.612756451 (518h29m56.483839486s) [noise=true , address=77.90.31.11:4556     , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.663674767 (518h29m56.534757813s) [noise=true , address=45.91.116.10:4084    , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.663704895 (518h29m56.53478794s) [seg_pos={0 0}, upload=454908]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:58:56.663995689 (518h29m56.535078754s) [noise=true , address=77.90.31.19:4842     , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection
Dec 27 05:58:56.664001861 (518h29m56.535084906s) [noise=true , address=77.90.31.19:4842     , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold
Dec 27 05:58:56.664126127 (518h29m56.535209172s) [noise=true , address=45.91.116.12:4226    , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 27 05:58:56.664169029 (518h29m56.535252065s) [noise=true , address=45.91.116.12:4226    , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold
Dec 27 05:58:56.664391382 (518h29m56.535474418s) [noise=true , address=77.90.31.21:4877     , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 27 05:58:56.664398546 (518h29m56.535481591s) [noise=true , address=77.90.31.21:4877     , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold
Dec 27 05:58:56.664419476 (518h29m56.535502521s) [seg_pos={0 0}, upload=454908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:58:56.664503536 (518h29m56.535586571s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:58:56.664509056 (518h29m56.535592091s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:58:56.958441263 (518h29m56.829524308s) [upload=454908]: Done uploading segments...
Dec 27 05:58:56.958480828 (518h29m56.829563863s) [upload=454908]: Upload finished. err: <nil>
Duration: 6.130888188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............X.............X................................]
duration: 
41.562374838s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:59:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 41.562430123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X.....X..X.XX]
duration: 
30.00257772s
trace: 
https://jaeger.dev.storj.io/trace/6e7b1f1cd9efe572
FAIL (2025-12-27 05:59:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA) context deadline exceeded
======LOGS======
 05:58:54.027618010 (518h29m53.898701045s) [upload=454908]: Starting upload
Dec 27 05:58:54.027674568 (518h29m53.898757613s) [upload=454908]: Done starting upload
Dec 27 05:58:54.027752656 (518h29m53.898835691s) [upload=454908]: Uploading segments...
Dec 27 05:58:54.027867967 (518h29m53.898951002s) [upload=454908]: Got next segment. Inline: false
Dec 27 05:58:54.027959360 (518h29m53.899042395s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:58:54.027966052 (518h29m53.899049088s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:58:54.273197646 (518h29m54.144280681s) [seg_pos={0 0}, upload=454908]: Begin upload segment...
Dec 27 05:58:54.273253432 (518h29m54.144336467s) [seg_pos={0 0}, upload=454908]: Done begin upload segment.
Dec 27 05:58:54.273335729 (518h29m54.144418764s) [noise=true , address=145.79.227.6:4259    , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.273355857 (518h29m54.144438902s) [noise=true , address=145.79.227.31:4797   , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.274397771 (518h29m54.145480816s) [noise=true , address=45.91.116.7:4009     , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.275285933 (518h29m54.146368968s) [noise=true , address=145.79.227.38:4978   , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.275978290 (518h29m54.147061335s) [noise=true , address=77.90.31.11:4556     , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.277650785 (518h29m54.14873382s) [noise=true , address=77.90.31.21:4877     , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.281356081 (518h29m54.152439116s) [noise=true , address=145.79.227.45:4231   , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.283248223 (518h29m54.154331268s) [noise=true , address=145.79.227.39:4013   , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.286543088 (518h29m54.157626123s) [noise=true , address=45.91.116.10:4084    , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.296837890 (518h29m54.167920935s) [noise=true , address=145.79.227.17:4971   , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.297626512 (518h29m54.168709547s) [noise=true , address=145.79.227.28:4725   , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.298856263 (518h29m54.169939298s) [noise=true , address=77.90.31.19:4842     , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.299710430 (518h29m54.170793485s) [noise=true , address=77.90.31.14:4661     , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.300773284 (518h29m54.171856319s) [noise=true , address=45.91.116.12:4226    , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.301333490 (518h29m54.172416535s) [noise=true , address=145.79.227.50:4374   , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:54.302620932 (518h29m54.173703967s) [noise=true , address=145.79.227.46:4603   , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Uploading piece...
Dec 27 05:58:55.915655523 (518h29m55.786738558s) [upload=454908]: Next returned nil segment
Dec 27 05:58:55.915665622 (518h29m55.786748657s) [upload=454908]: Waiting for error group managing segments...
Dec 27 05:58:56.302185873 (518h29m56.173268918s) [noise=true , address=145.79.227.28:4725   , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.305487398 (518h29m56.176570434s) [noise=true , address=145.79.227.45:4231   , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.305946472 (518h29m56.177029507s) [noise=true , address=145.79.227.17:4971   , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.310843968 (518h29m56.181927003s) [noise=true , address=145.79.227.50:4374   , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.313521086 (518h29m56.184604141s) [noise=true , address=145.79.227.38:4978   , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.317048212 (518h29m56.188131247s) [noise=true , address=145.79.227.39:4013   , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.317491997 (518h29m56.188575043s) [noise=true , address=145.79.227.31:4797   , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.327595456 (518h29m56.198678491s) [noise=true , address=145.79.227.6:4259    , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.332749460 (518h29m56.203832505s) [noise=true , address=145.79.227.46:4603   , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.467173567 (518h29m56.338256602s) [noise=true , address=77.90.31.14:4661     , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.555793674 (518h29m56.426876719s) [noise=true , address=45.91.116.7:4009     , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.612756451 (518h29m56.483839486s) [noise=true , address=77.90.31.11:4556     , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.663674767 (518h29m56.534757813s) [noise=true , address=45.91.116.10:4084    , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil>
Dec 27 05:58:56.663704895 (518h29m56.53478794s) [seg_pos={0 0}, upload=454908]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:58:56.663995689 (518h29m56.535078754s) [noise=true , address=77.90.31.19:4842     , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection
Dec 27 05:58:56.664001861 (518h29m56.535084906s) [noise=true , address=77.90.31.19:4842     , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold
Dec 27 05:58:56.664126127 (518h29m56.535209172s) [noise=true , address=45.91.116.12:4226    , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 27 05:58:56.664169029 (518h29m56.535252065s) [noise=true , address=45.91.116.12:4226    , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold
Dec 27 05:58:56.664391382 (518h29m56.535474418s) [noise=true , address=77.90.31.21:4877     , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 27 05:58:56.664398546 (518h29m56.535481591s) [noise=true , address=77.90.31.21:4877     , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold
Dec 27 05:58:56.664419476 (518h29m56.535502521s) [seg_pos={0 0}, upload=454908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:58:56.664503536 (518h29m56.535586571s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:58:56.664509056 (518h29m56.535592091s) [upload=454908]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:58:56.958441263 (518h29m56.829524308s) [upload=454908]: Done uploading segments...
Dec 27 05:58:56.958480828 (518h29m56.829563863s) [upload=454908]: Upload finished. err: <nil>
Duration: 30.002612936s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...........................................X................]
duration: 
3.867159625s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:59:12 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.867191216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [X.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X.....X..X.X]
duration: 
30.003078439s
trace: 
https://jaeger.dev.storj.io/trace/59960c54e72c9931
FAIL (2025-12-27 05:58:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 05:58:16.582138792 (518h29m16.453221837s) [upload=454900]: Starting upload
Dec 27 05:58:16.582740208 (518h29m16.453823263s) [upload=454900]: Done starting upload
Dec 27 05:58:16.583243205 (518h29m16.454326251s) [upload=454900]: Uploading segments...
Dec 27 05:58:16.583449238 (518h29m16.454532273s) [upload=454900]: Got next segment. Inline: false
Dec 27 05:58:16.583529901 (518h29m16.454612936s) [upload=454900]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:58:16.583545381 (518h29m16.454628416s) [upload=454900]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:58:16.833916570 (518h29m16.704999615s) [seg_pos={0 0}, upload=454900]: Begin upload segment...
Dec 27 05:58:16.833971384 (518h29m16.705054419s) [seg_pos={0 0}, upload=454900]: Done begin upload segment.
Dec 27 05:58:16.834001351 (518h29m16.705084396s) [noise=true , address=145.79.227.36:4519   , piece_id=5HJG2WKIVDYMPT7HOZ7WUKXGDB77JPOTHHYEHXVUJ6CGUZB4R26Q, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.834113153 (518h29m16.705196199s) [noise=true , address=145.79.227.11:4731   , piece_id=DP7XDYVZK55UK53D6E4EW2PQNP6QGB25TG2NZO3UPLP6RADWKQNQ, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.834808978 (518h29m16.705892024s) [noise=true , address=145.79.227.49:4338   , piece_id=3ROH22XWEUYJDRLJ2DFX4PAGS3Z3S4UOWNB6SF5ZS6UPLL5UYOWA, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.835482842 (518h29m16.706565887s) [noise=true , address=145.79.227.41:4085   , piece_id=QVLVSA37STLCW45VQ3BAZB2HMJFQ63D6YEO3KAHUHFN4S3UE233Q, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.835669597 (518h29m16.706752643s) [noise=true , address=45.91.116.7:4009     , piece_id=X5AR7IKV7ZO5UOOV4WWGRCWM2FE6NSZZYN4QUDIFOQYBWUAWYAQQ, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.835996189 (518h29m16.707079234s) [noise=true , address=77.90.31.11:4553     , piece_id=VVSTPOZ736JILQKUPZN675CTLJJOXK4XOFQJRGXQZVUVMTVPRE3A, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.836350815 (518h29m16.70743386s) [noise=true , address=45.91.116.14:4192    , piece_id=NLDJW6EQGKGLDFRPBFJEXE7CD27QUPNS73NIZP4NQHXKLP5ZLSWA, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.836423523 (518h29m16.707506569s) [noise=true , address=45.91.116.10:4081    , piece_id=RJ4PXLNMOZLP5XY55VY7HXYEOV7KG3KK6RVPQ7E6GRJPRS3KGCPA, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.836761667 (518h29m16.707844702s) [noise=true , address=145.79.227.37:4566   , piece_id=KLPZWG65YY6ZSYVZVIIUXGBOKBCC2RFMIEUWRGTUJIBLJKEEK3KA, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.836896694 (518h29m16.707979749s) [noise=true , address=77.90.31.14:4661     , piece_id=2ZMZWZOLEZDMQ4SWK6SVFBEENAZ3AFYQV5CVI2BP2X3CQ4XFXZIQ, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.837366469 (518h29m16.708449504s) [noise=true , address=45.91.116.6:4975     , piece_id=KCE6RAAFCFO2HYAIKXVV525VZFGL2UIQ6KSTN5POTIYJBZ6NCEZA, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.837569736 (518h29m16.708652781s) [noise=true , address=145.79.227.28:4725   , piece_id=VQYLRXNEZOAYTRAPVBTE2GTJNXA2PRMLB5TGYV3YUWTJX543CQWA, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.837865890 (518h29m16.708948945s) [noise=true , address=45.91.116.9:4263     , piece_id=NHSJZGA56YHHIWEPZ4X4R53JZ74KQDCJILMJUZWGWF645GS4GOZA, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.838225956 (518h29m16.709308991s) [noise=true , address=145.79.227.47:4267   , piece_id=FL5J62PFL75ASR7JM5PDHZ4EOXMT5FVGYMZDPZ2VCI2B2PPLUTMA, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.838621749 (518h29m16.709704784s) [noise=true , address=45.91.116.12:4226    , piece_id=DJPNF2OIFTLGVK7IONU2CJEHFV25WM7FGWUCSTA2PAVANPY5WOKQ, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:16.839183839 (518h29m16.710266884s) [noise=true , address=77.90.31.15:4700     , piece_id=2KNS6NLYH25NKKUSDNO4DU25QOQFJDPPQWZEF2LJ6BKJ5KOGCDHQ, seg_pos={0 0}, upload=454900]: Uploading piece...
Dec 27 05:58:18.545144847 (518h29m18.416227882s) [upload=454900]: Next returned nil segment
Dec 27 05:58:18.545162821 (518h29m18.416245866s) [upload=454900]: Waiting for error group managing segments...
Dec 27 05:58:18.829104414 (518h29m18.700187459s) [noise=true , address=145.79.227.37:4566   , piece_id=KLPZWG65YY6ZSYVZVIIUXGBOKBCC2RFMIEUWRGTUJIBLJKEEK3KA, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:18.841949603 (518h29m18.713032638s) [noise=true , address=145.79.227.36:4519   , piece_id=5HJG2WKIVDYMPT7HOZ7WUKXGDB77JPOTHHYEHXVUJ6CGUZB4R26Q, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:18.848614224 (518h29m18.719697259s) [noise=true , address=145.79.227.41:4085   , piece_id=QVLVSA37STLCW45VQ3BAZB2HMJFQ63D6YEO3KAHUHFN4S3UE233Q, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:18.848714185 (518h29m18.71979722s) [noise=true , address=145.79.227.11:4731   , piece_id=DP7XDYVZK55UK53D6E4EW2PQNP6QGB25TG2NZO3UPLP6RADWKQNQ, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:18.851076413 (518h29m18.722159448s) [noise=true , address=145.79.227.28:4725   , piece_id=VQYLRXNEZOAYTRAPVBTE2GTJNXA2PRMLB5TGYV3YUWTJX543CQWA, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:18.852441232 (518h29m18.723524267s) [noise=true , address=145.79.227.47:4267   , piece_id=FL5J62PFL75ASR7JM5PDHZ4EOXMT5FVGYMZDPZ2VCI2B2PPLUTMA, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:18.885158229 (518h29m18.756241275s) [noise=true , address=145.79.227.49:4338   , piece_id=3ROH22XWEUYJDRLJ2DFX4PAGS3Z3S4UOWNB6SF5ZS6UPLL5UYOWA, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:19.073622106 (518h29m18.944705141s) [noise=true , address=77.90.31.15:4700     , piece_id=2KNS6NLYH25NKKUSDNO4DU25QOQFJDPPQWZEF2LJ6BKJ5KOGCDHQ, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:19.088910378 (518h29m18.959993413s) [noise=true , address=45.91.116.9:4263     , piece_id=NHSJZGA56YHHIWEPZ4X4R53JZ74KQDCJILMJUZWGWF645GS4GOZA, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:19.102428138 (518h29m18.973511194s) [noise=true , address=45.91.116.7:4009     , piece_id=X5AR7IKV7ZO5UOOV4WWGRCWM2FE6NSZZYN4QUDIFOQYBWUAWYAQQ, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:19.130020328 (518h29m19.001103363s) [noise=true , address=45.91.116.10:4081    , piece_id=RJ4PXLNMOZLP5XY55VY7HXYEOV7KG3KK6RVPQ7E6GRJPRS3KGCPA, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:19.151488787 (518h29m19.022571822s) [noise=true , address=45.91.116.14:4192    , piece_id=NLDJW6EQGKGLDFRPBFJEXE7CD27QUPNS73NIZP4NQHXKLP5ZLSWA, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:19.374996263 (518h29m19.246079308s) [noise=true , address=45.91.116.12:4226    , piece_id=DJPNF2OIFTLGVK7IONU2CJEHFV25WM7FGWUCSTA2PAVANPY5WOKQ, seg_pos={0 0}, upload=454900]: Done uploading piece. err: <nil>
Dec 27 05:58:19.375019117 (518h29m19.246102162s) [seg_pos={0 0}, upload=454900]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:58:19.375221341 (518h29m19.246304376s) [noise=true , address=77.90.31.14:4661     , piece_id=2ZMZWZOLEZDMQ4SWK6SVFBEENAZ3AFYQV5CVI2BP2X3CQ4XFXZIQ, seg_pos={0 0}, upload=454900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 27 05:58:19.375228355 (518h29m19.246311401s) [noise=true , address=77.90.31.14:4661     , piece_id=2ZMZWZOLEZDMQ4SWK6SVFBEENAZ3AFYQV5CVI2BP2X3CQ4XFXZIQ, seg_pos={0 0}, upload=454900]: Long tail cancellation due to optimal threshold
Dec 27 05:58:19.375472831 (518h29m19.246555876s) [noise=true , address=77.90.31.11:4553     , piece_id=VVSTPOZ736JILQKUPZN675CTLJJOXK4XOFQJRGXQZVUVMTVPRE3A, seg_pos={0 0}, upload=454900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF
Dec 27 05:58:19.375478932 (518h29m19.246561978s) [noise=true , address=77.90.31.11:4553     , piece_id=VVSTPOZ736JILQKUPZN675CTLJJOXK4XOFQJRGXQZVUVMTVPRE3A, seg_pos={0 0}, upload=454900]: Long tail cancellation due to optimal threshold
Dec 27 05:58:19.375596686 (518h29m19.246679721s) [noise=true , address=45.91.116.6:4975     , piece_id=KCE6RAAFCFO2HYAIKXVV525VZFGL2UIQ6KSTN5POTIYJBZ6NCEZA, seg_pos={0 0}, upload=454900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:55114->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:55114->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55114->45.91.116.6:4975: use of closed network connection
Dec 27 05:58:19.375604080 (518h29m19.246687125s) [noise=true , address=45.91.116.6:4975     , piece_id=KCE6RAAFCFO2HYAIKXVV525VZFGL2UIQ6KSTN5POTIYJBZ6NCEZA, seg_pos={0 0}, upload=454900]: Long tail cancellation due to optimal threshold
Dec 27 05:58:19.375620582 (518h29m19.246703617s) [seg_pos={0 0}, upload=454900]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:58:19.375693270 (518h29m19.246776315s) [upload=454900]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:58:19.375698300 (518h29m19.246781335s) [upload=454900]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:58:19.669194927 (518h29m19.540277962s) [upload=454900]: Done uploading segments...
Dec 27 05:58:19.669216688 (518h29m19.540299723s) [upload=454900]: Upload finished. err: <nil>
Duration: 30.003102855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..............X.............X...............................]
duration: 
15.443801839s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:58:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 15.443830263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [............................................X...............]
duration: 
3.790754689s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:58:00 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.790783493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............X.............X..............................]
duration: 
43.762133816s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:57:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 43.762153985s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X.....X..X.]
duration: 
5.883453791s
trace: 
https://jaeger.dev.storj.io/trace/57b58a50149e9a93
FAIL (2025-12-27 05:57:12 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:55:58.057275132 (518h26m57.928358167s) [upload=454867]: Starting upload
Dec 27 05:55:58.057370304 (518h26m57.928453369s) [upload=454867]: Done starting upload
Dec 27 05:55:58.057487556 (518h26m57.928570612s) [upload=454867]: Uploading segments...
Dec 27 05:55:58.057597626 (518h26m57.928680681s) [upload=454867]: Got next segment. Inline: false
Dec 27 05:55:58.057646279 (518h26m57.928729314s) [upload=454867]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:55:58.057654014 (518h26m57.928737069s) [upload=454867]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:55:58.303387672 (518h26m58.174470707s) [seg_pos={0 0}, upload=454867]: Begin upload segment...
Dec 27 05:55:58.303439801 (518h26m58.174522836s) [seg_pos={0 0}, upload=454867]: Done begin upload segment.
Dec 27 05:55:58.303489184 (518h26m58.17457222s) [noise=true , address=145.79.227.51:4410   , piece_id=AO5KZFCIZENZCMLM67DBKR75KNQJ2M6ZD3YCYS3W3KEUMME6OOGQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.303510435 (518h26m58.17459347s) [noise=true , address=45.91.116.9:4261     , piece_id=4HJACKQP4DD53KX4WGPBDDOD6FQPO5BJQ5LWIROTWKTC73Y56WSQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.304311490 (518h26m58.175394556s) [noise=true , address=145.79.227.28:4726   , piece_id=FV5AL53TTV57ZZXF65X32US7NBCYPHMWYJTOMBRU7J5QBRWHJM5Q, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.305184733 (518h26m58.176267768s) [noise=true , address=77.90.31.21:4877     , piece_id=QI45ZCUOJ64XVCEH6B6VIRLNLK5364COV4K6Z2CXIMMM72ANAPHA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.305617558 (518h26m58.176700593s) [noise=true , address=145.79.227.37:4565   , piece_id=YERK2VMRVFB3Y4JV2IQUCHLLCUA5RDVOGVCF3BDGOY25JQZTMG2A, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.305995528 (518h26m58.177078563s) [noise=true , address=77.90.31.13:4629     , piece_id=YN4J25T6YCXUYB63GX5JX24OCCY2NFO6DR7KB3CRS3PZCPUQJNMQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.306399797 (518h26m58.177482832s) [noise=true , address=45.91.116.4:4939     , piece_id=7P4HNDPGB4BSKVEYSCPLBPIOXSUHRPIF4IL4IULI3MQRQSADXLLQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.306855675 (518h26m58.17793871s) [noise=true , address=145.79.227.11:4729   , piece_id=UKHK53M7KOQOQTKPPKYWZ23PZ6DSH4FE45HITETRRM7WCPTWRZSQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.308215925 (518h26m58.17929896s) [noise=true , address=145.79.227.36:4517   , piece_id=4L3DVQQHR3JMCFCQZW7FF3NSYRP54IJH7SSAU45DOVBXCYH5RQKA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.309403226 (518h26m58.180486261s) [noise=true , address=145.79.227.41:4085   , piece_id=SG4FK6L2OU6KI6HAX63WJJ4E2CHLZIA2VWKXVNEMQYVE6L4QXXVA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.310552574 (518h26m58.181635609s) [noise=true , address=145.79.227.9:4366    , piece_id=XHFYF5YWALXC5DQGZJWWJQ7K357AHDZOTFWIZJS2MJ3JZ7A4CORA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.310998814 (518h26m58.182081849s) [noise=true , address=45.91.116.14:4192    , piece_id=NMVV4XQ4JWLKOFVFUULMJPRGYMOVZ6XZ6WDC7ZN2MPSQNRVYWTZA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.312197277 (518h26m58.183280312s) [noise=true , address=145.79.227.39:4014   , piece_id=4FVX5QZDSELSBVXIPNCHXBT5FMGPWCQ6QPMU3Y3IUTJI7AHDS6HQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.312632245 (518h26m58.18371528s) [noise=true , address=145.79.227.19:4077   , piece_id=CT2AMQWM5CKFOF7K6DQWYXL4IYBEM7C4IXX376QTOW7MRPWJLROA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.314119557 (518h26m58.185202592s) [noise=true , address=145.79.227.38:4978   , piece_id=DRKEEAXCYENHB64UPUV5DFJMT3NBDCLIJ5W4R62U423YJI5J2XGA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.315596068 (518h26m58.186679103s) [noise=true , address=145.79.227.49:4339   , piece_id=BFP5FXRPHS3QB4CP2XC67WZG3EUQZIHPNWILWADLSB3RBWJ7OLIQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:56:00.010272398 (518h26m59.881355443s) [upload=454867]: Next returned nil segment
Dec 27 05:56:00.010289760 (518h26m59.881372795s) [upload=454867]: Waiting for error group managing segments...
Dec 27 05:56:00.326212200 (518h27m0.197295235s) [noise=true , address=145.79.227.41:4085   , piece_id=SG4FK6L2OU6KI6HAX63WJJ4E2CHLZIA2VWKXVNEMQYVE6L4QXXVA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.341741120 (518h27m0.212824165s) [noise=true , address=145.79.227.19:4077   , piece_id=CT2AMQWM5CKFOF7K6DQWYXL4IYBEM7C4IXX376QTOW7MRPWJLROA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.377972711 (518h27m0.249055756s) [noise=true , address=145.79.227.28:4726   , piece_id=FV5AL53TTV57ZZXF65X32US7NBCYPHMWYJTOMBRU7J5QBRWHJM5Q, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.378535242 (518h27m0.249618277s) [noise=true , address=145.79.227.9:4366    , piece_id=XHFYF5YWALXC5DQGZJWWJQ7K357AHDZOTFWIZJS2MJ3JZ7A4CORA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.380810754 (518h27m0.251893799s) [noise=true , address=145.79.227.38:4978   , piece_id=DRKEEAXCYENHB64UPUV5DFJMT3NBDCLIJ5W4R62U423YJI5J2XGA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.381401690 (518h27m0.252484735s) [noise=true , address=145.79.227.49:4339   , piece_id=BFP5FXRPHS3QB4CP2XC67WZG3EUQZIHPNWILWADLSB3RBWJ7OLIQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.386810500 (518h27m0.257893545s) [noise=true , address=145.79.227.39:4014   , piece_id=4FVX5QZDSELSBVXIPNCHXBT5FMGPWCQ6QPMU3Y3IUTJI7AHDS6HQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.387825252 (518h27m0.258908297s) [noise=true , address=145.79.227.36:4517   , piece_id=4L3DVQQHR3JMCFCQZW7FF3NSYRP54IJH7SSAU45DOVBXCYH5RQKA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.388517610 (518h27m0.259600645s) [noise=true , address=145.79.227.11:4729   , piece_id=UKHK53M7KOQOQTKPPKYWZ23PZ6DSH4FE45HITETRRM7WCPTWRZSQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.388982966 (518h27m0.260066011s) [noise=true , address=145.79.227.37:4565   , piece_id=YERK2VMRVFB3Y4JV2IQUCHLLCUA5RDVOGVCF3BDGOY25JQZTMG2A, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.389461107 (518h27m0.260544152s) [noise=true , address=145.79.227.51:4410   , piece_id=AO5KZFCIZENZCMLM67DBKR75KNQJ2M6ZD3YCYS3W3KEUMME6OOGQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.622286185 (518h27m0.493369231s) [noise=true , address=77.90.31.13:4629     , piece_id=YN4J25T6YCXUYB63GX5JX24OCCY2NFO6DR7KB3CRS3PZCPUQJNMQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:01.240694859 (518h27m1.111777905s) [noise=true , address=45.91.116.9:4261     , piece_id=4HJACKQP4DD53KX4WGPBDDOD6FQPO5BJQ5LWIROTWKTC73Y56WSQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:01.240720779 (518h27m1.111803824s) [seg_pos={0 0}, upload=454867]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:56:01.240951328 (518h27m1.112034374s) [noise=true , address=77.90.31.21:4877     , piece_id=QI45ZCUOJ64XVCEH6B6VIRLNLK5364COV4K6Z2CXIMMM72ANAPHA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 27 05:56:01.240961618 (518h27m1.112044663s) [noise=true , address=77.90.31.21:4877     , piece_id=QI45ZCUOJ64XVCEH6B6VIRLNLK5364COV4K6Z2CXIMMM72ANAPHA, seg_pos={0 0}, upload=454867]: Long tail cancellation due to optimal threshold
Dec 27 05:56:01.241594032 (518h27m1.112677077s) [noise=true , address=45.91.116.4:4939     , piece_id=7P4HNDPGB4BSKVEYSCPLBPIOXSUHRPIF4IL4IULI3MQRQSADXLLQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 05:56:01.241603661 (518h27m1.112686706s) [noise=true , address=45.91.116.4:4939     , piece_id=7P4HNDPGB4BSKVEYSCPLBPIOXSUHRPIF4IL4IULI3MQRQSADXLLQ, seg_pos={0 0}, upload=454867]: Long tail cancellation due to optimal threshold
Dec 27 05:56:01.241757103 (518h27m1.112840148s) [noise=true , address=45.91.116.14:4192    , piece_id=NMVV4XQ4JWLKOFVFUULMJPRGYMOVZ6XZ6WDC7ZN2MPSQNRVYWTZA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 27 05:56:01.241764517 (518h27m1.112847572s) [noise=true , address=45.91.116.14:4192    , piece_id=NMVV4XQ4JWLKOFVFUULMJPRGYMOVZ6XZ6WDC7ZN2MPSQNRVYWTZA, seg_pos={0 0}, upload=454867]: Long tail cancellation due to optimal threshold
Dec 27 05:56:01.241786869 (518h27m1.112869915s) [seg_pos={0 0}, upload=454867]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:56:01.242341756 (518h27m1.113424811s) [upload=454867]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:56:01.242351534 (518h27m1.11343459s) [upload=454867]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:56:01.538302414 (518h27m1.409385469s) [upload=454867]: Done uploading segments...
Dec 27 05:56:01.538327382 (518h27m1.409410427s) [upload=454867]: Upload finished. err: <nil>
Duration: 5.883485381s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.............................................X..............]
duration: 
3.59565762s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:56:52 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.59568293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [................X.............X.............................]
duration: 
37.069317685s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:56:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 37.069340238s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [.XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X.....X..X]
duration: 
30.002603927s
trace: 
https://jaeger.dev.storj.io/trace/57b58a50149e9a93
FAIL (2025-12-27 05:56:27 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:55:58.057275132 (518h26m57.928358167s) [upload=454867]: Starting upload
Dec 27 05:55:58.057370304 (518h26m57.928453369s) [upload=454867]: Done starting upload
Dec 27 05:55:58.057487556 (518h26m57.928570612s) [upload=454867]: Uploading segments...
Dec 27 05:55:58.057597626 (518h26m57.928680681s) [upload=454867]: Got next segment. Inline: false
Dec 27 05:55:58.057646279 (518h26m57.928729314s) [upload=454867]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:55:58.057654014 (518h26m57.928737069s) [upload=454867]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:55:58.303387672 (518h26m58.174470707s) [seg_pos={0 0}, upload=454867]: Begin upload segment...
Dec 27 05:55:58.303439801 (518h26m58.174522836s) [seg_pos={0 0}, upload=454867]: Done begin upload segment.
Dec 27 05:55:58.303489184 (518h26m58.17457222s) [noise=true , address=145.79.227.51:4410   , piece_id=AO5KZFCIZENZCMLM67DBKR75KNQJ2M6ZD3YCYS3W3KEUMME6OOGQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.303510435 (518h26m58.17459347s) [noise=true , address=45.91.116.9:4261     , piece_id=4HJACKQP4DD53KX4WGPBDDOD6FQPO5BJQ5LWIROTWKTC73Y56WSQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.304311490 (518h26m58.175394556s) [noise=true , address=145.79.227.28:4726   , piece_id=FV5AL53TTV57ZZXF65X32US7NBCYPHMWYJTOMBRU7J5QBRWHJM5Q, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.305184733 (518h26m58.176267768s) [noise=true , address=77.90.31.21:4877     , piece_id=QI45ZCUOJ64XVCEH6B6VIRLNLK5364COV4K6Z2CXIMMM72ANAPHA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.305617558 (518h26m58.176700593s) [noise=true , address=145.79.227.37:4565   , piece_id=YERK2VMRVFB3Y4JV2IQUCHLLCUA5RDVOGVCF3BDGOY25JQZTMG2A, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.305995528 (518h26m58.177078563s) [noise=true , address=77.90.31.13:4629     , piece_id=YN4J25T6YCXUYB63GX5JX24OCCY2NFO6DR7KB3CRS3PZCPUQJNMQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.306399797 (518h26m58.177482832s) [noise=true , address=45.91.116.4:4939     , piece_id=7P4HNDPGB4BSKVEYSCPLBPIOXSUHRPIF4IL4IULI3MQRQSADXLLQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.306855675 (518h26m58.17793871s) [noise=true , address=145.79.227.11:4729   , piece_id=UKHK53M7KOQOQTKPPKYWZ23PZ6DSH4FE45HITETRRM7WCPTWRZSQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.308215925 (518h26m58.17929896s) [noise=true , address=145.79.227.36:4517   , piece_id=4L3DVQQHR3JMCFCQZW7FF3NSYRP54IJH7SSAU45DOVBXCYH5RQKA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.309403226 (518h26m58.180486261s) [noise=true , address=145.79.227.41:4085   , piece_id=SG4FK6L2OU6KI6HAX63WJJ4E2CHLZIA2VWKXVNEMQYVE6L4QXXVA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.310552574 (518h26m58.181635609s) [noise=true , address=145.79.227.9:4366    , piece_id=XHFYF5YWALXC5DQGZJWWJQ7K357AHDZOTFWIZJS2MJ3JZ7A4CORA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.310998814 (518h26m58.182081849s) [noise=true , address=45.91.116.14:4192    , piece_id=NMVV4XQ4JWLKOFVFUULMJPRGYMOVZ6XZ6WDC7ZN2MPSQNRVYWTZA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.312197277 (518h26m58.183280312s) [noise=true , address=145.79.227.39:4014   , piece_id=4FVX5QZDSELSBVXIPNCHXBT5FMGPWCQ6QPMU3Y3IUTJI7AHDS6HQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.312632245 (518h26m58.18371528s) [noise=true , address=145.79.227.19:4077   , piece_id=CT2AMQWM5CKFOF7K6DQWYXL4IYBEM7C4IXX376QTOW7MRPWJLROA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.314119557 (518h26m58.185202592s) [noise=true , address=145.79.227.38:4978   , piece_id=DRKEEAXCYENHB64UPUV5DFJMT3NBDCLIJ5W4R62U423YJI5J2XGA, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:55:58.315596068 (518h26m58.186679103s) [noise=true , address=145.79.227.49:4339   , piece_id=BFP5FXRPHS3QB4CP2XC67WZG3EUQZIHPNWILWADLSB3RBWJ7OLIQ, seg_pos={0 0}, upload=454867]: Uploading piece...
Dec 27 05:56:00.010272398 (518h26m59.881355443s) [upload=454867]: Next returned nil segment
Dec 27 05:56:00.010289760 (518h26m59.881372795s) [upload=454867]: Waiting for error group managing segments...
Dec 27 05:56:00.326212200 (518h27m0.197295235s) [noise=true , address=145.79.227.41:4085   , piece_id=SG4FK6L2OU6KI6HAX63WJJ4E2CHLZIA2VWKXVNEMQYVE6L4QXXVA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.341741120 (518h27m0.212824165s) [noise=true , address=145.79.227.19:4077   , piece_id=CT2AMQWM5CKFOF7K6DQWYXL4IYBEM7C4IXX376QTOW7MRPWJLROA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.377972711 (518h27m0.249055756s) [noise=true , address=145.79.227.28:4726   , piece_id=FV5AL53TTV57ZZXF65X32US7NBCYPHMWYJTOMBRU7J5QBRWHJM5Q, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.378535242 (518h27m0.249618277s) [noise=true , address=145.79.227.9:4366    , piece_id=XHFYF5YWALXC5DQGZJWWJQ7K357AHDZOTFWIZJS2MJ3JZ7A4CORA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.380810754 (518h27m0.251893799s) [noise=true , address=145.79.227.38:4978   , piece_id=DRKEEAXCYENHB64UPUV5DFJMT3NBDCLIJ5W4R62U423YJI5J2XGA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.381401690 (518h27m0.252484735s) [noise=true , address=145.79.227.49:4339   , piece_id=BFP5FXRPHS3QB4CP2XC67WZG3EUQZIHPNWILWADLSB3RBWJ7OLIQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.386810500 (518h27m0.257893545s) [noise=true , address=145.79.227.39:4014   , piece_id=4FVX5QZDSELSBVXIPNCHXBT5FMGPWCQ6QPMU3Y3IUTJI7AHDS6HQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.387825252 (518h27m0.258908297s) [noise=true , address=145.79.227.36:4517   , piece_id=4L3DVQQHR3JMCFCQZW7FF3NSYRP54IJH7SSAU45DOVBXCYH5RQKA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.388517610 (518h27m0.259600645s) [noise=true , address=145.79.227.11:4729   , piece_id=UKHK53M7KOQOQTKPPKYWZ23PZ6DSH4FE45HITETRRM7WCPTWRZSQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.388982966 (518h27m0.260066011s) [noise=true , address=145.79.227.37:4565   , piece_id=YERK2VMRVFB3Y4JV2IQUCHLLCUA5RDVOGVCF3BDGOY25JQZTMG2A, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.389461107 (518h27m0.260544152s) [noise=true , address=145.79.227.51:4410   , piece_id=AO5KZFCIZENZCMLM67DBKR75KNQJ2M6ZD3YCYS3W3KEUMME6OOGQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:00.622286185 (518h27m0.493369231s) [noise=true , address=77.90.31.13:4629     , piece_id=YN4J25T6YCXUYB63GX5JX24OCCY2NFO6DR7KB3CRS3PZCPUQJNMQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:01.240694859 (518h27m1.111777905s) [noise=true , address=45.91.116.9:4261     , piece_id=4HJACKQP4DD53KX4WGPBDDOD6FQPO5BJQ5LWIROTWKTC73Y56WSQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: <nil>
Dec 27 05:56:01.240720779 (518h27m1.111803824s) [seg_pos={0 0}, upload=454867]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:56:01.240951328 (518h27m1.112034374s) [noise=true , address=77.90.31.21:4877     , piece_id=QI45ZCUOJ64XVCEH6B6VIRLNLK5364COV4K6Z2CXIMMM72ANAPHA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 27 05:56:01.240961618 (518h27m1.112044663s) [noise=true , address=77.90.31.21:4877     , piece_id=QI45ZCUOJ64XVCEH6B6VIRLNLK5364COV4K6Z2CXIMMM72ANAPHA, seg_pos={0 0}, upload=454867]: Long tail cancellation due to optimal threshold
Dec 27 05:56:01.241594032 (518h27m1.112677077s) [noise=true , address=45.91.116.4:4939     , piece_id=7P4HNDPGB4BSKVEYSCPLBPIOXSUHRPIF4IL4IULI3MQRQSADXLLQ, seg_pos={0 0}, upload=454867]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 05:56:01.241603661 (518h27m1.112686706s) [noise=true , address=45.91.116.4:4939     , piece_id=7P4HNDPGB4BSKVEYSCPLBPIOXSUHRPIF4IL4IULI3MQRQSADXLLQ, seg_pos={0 0}, upload=454867]: Long tail cancellation due to optimal threshold
Dec 27 05:56:01.241757103 (518h27m1.112840148s) [noise=true , address=45.91.116.14:4192    , piece_id=NMVV4XQ4JWLKOFVFUULMJPRGYMOVZ6XZ6WDC7ZN2MPSQNRVYWTZA, seg_pos={0 0}, upload=454867]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 27 05:56:01.241764517 (518h27m1.112847572s) [noise=true , address=45.91.116.14:4192    , piece_id=NMVV4XQ4JWLKOFVFUULMJPRGYMOVZ6XZ6WDC7ZN2MPSQNRVYWTZA, seg_pos={0 0}, upload=454867]: Long tail cancellation due to optimal threshold
Dec 27 05:56:01.241786869 (518h27m1.112869915s) [seg_pos={0 0}, upload=454867]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:56:01.242341756 (518h27m1.113424811s) [upload=454867]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:56:01.242351534 (518h27m1.11343459s) [upload=454867]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:56:01.538302414 (518h27m1.409385469s) [upload=454867]: Done uploading segments...
Dec 27 05:56:01.538327382 (518h27m1.409410427s) [upload=454867]: Upload finished. err: <nil>
Duration: 30.002625668s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..............................................X.............]
duration: 
3.75648902s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:56:26 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.756510991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.................X.............X............................]
duration: 
46.828047275s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:56:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 46.828068856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [..XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X.....X..]
duration: 
5.746942108s
trace: 
https://jaeger.dev.storj.io/trace/6e98bce886072abb
FAIL (2025-12-27 05:55:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:54:09.292640120 (518h25m9.163723175s) [upload=454841]: Starting upload
Dec 27 05:54:09.292721946 (518h25m9.163804992s) [upload=454841]: Done starting upload
Dec 27 05:54:09.292926666 (518h25m9.164009701s) [upload=454841]: Uploading segments...
Dec 27 05:54:09.292961653 (518h25m9.164044698s) [upload=454841]: Got next segment. Inline: false
Dec 27 05:54:09.292984676 (518h25m9.164067711s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:54:09.292991269 (518h25m9.164074304s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:54:09.541202385 (518h25m9.41228542s) [seg_pos={0 0}, upload=454841]: Begin upload segment...
Dec 27 05:54:09.541255896 (518h25m9.412338942s) [seg_pos={0 0}, upload=454841]: Done begin upload segment.
Dec 27 05:54:09.541280934 (518h25m9.412363969s) [noise=true , address=145.79.227.4:4185    , piece_id=Q4S6EOET5WVPB3YOIFAYKJFMDD2MXZHIK7FBYH2V3LOQEYLKCMTQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.541589151 (518h25m9.412672197s) [noise=true , address=145.79.227.38:4979   , piece_id=D67D5ENRB5DTU2NYU6VLYQMM763TLYOUCVD5K3IWB3U4UQLG47HA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.541717636 (518h25m9.412800682s) [noise=true , address=145.79.227.45:4229   , piece_id=4LMM6JU7AZLTYY5DE7DISC6CF3TNMV57QCBVKKDMZW477PVP3NUQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.542275909 (518h25m9.413358945s) [noise=true , address=145.79.227.6:4257    , piece_id=7QDSGX6NHIOIQL2MTABI3FWZSPN4UVROOVTOPZOXVURS4KEP5DSA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.542616096 (518h25m9.413699132s) [noise=true , address=77.90.31.10:4520     , piece_id=BBE2ZFKCBKJUU4O7CKQM7EMGUMPAXF3RJJZVY4BZK3XD2LBQM5QA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.542940395 (518h25m9.41402343s) [noise=true , address=145.79.227.43:4158   , piece_id=JLSZMH5VGAJRQIBR6N4C2LIBRFVAVTB6IU5UUTIW6RN3DH66OWSA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.543213785 (518h25m9.41429683s) [noise=true , address=45.91.116.12:4226    , piece_id=HAUZMVQJ5PVFSHJYB6B623MRPEDFXPAPOOIEHGGWS22RSZFTMURQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.543528014 (518h25m9.414611059s) [noise=true , address=145.79.227.9:4366    , piece_id=Q64OQVUCVW7F6F4ZLU7VNXG4BPWLDJGFLIMNG7MWECGSZ3D7QW5Q, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.543894992 (518h25m9.414978037s) [noise=true , address=145.79.227.48:4301   , piece_id=SPCBYJH2254DRQATPEVP5KR7DOM4CIDA7N5Y5CKRYOA2QAZAYDVA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.544295034 (518h25m9.415378069s) [noise=true , address=45.91.116.11:4118    , piece_id=DNJN7GRTCG5FNW4U7D5MV3TFZMSOUEJCBVZI7NJ6K3ZKLCWGKWVQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.544661251 (518h25m9.415744286s) [noise=true , address=145.79.227.42:4121   , piece_id=KVCHRVKHNP5C3ZGZN5SQLZ7PYFCWS433K2S2EWYDT42TWPA5D6KQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.545507773 (518h25m9.416590808s) [noise=true , address=145.79.227.26:4654   , piece_id=33Q6U3MJS3UUWPCGKUD62BL4GLB6WLYXJQQBR5QI3WCNFL3MOTOA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.546101915 (518h25m9.41718495s) [noise=true , address=45.91.116.13:4155    , piece_id=BAU2227SSFDURRDCFQKENLCS3DW7VGGOMOL5S2SMWYEBQ6FA6E7A, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.547363156 (518h25m9.418446191s) [noise=true , address=45.91.116.4:4939     , piece_id=FRTWR72DXMVKTEKPWSJALHLDAZUZ3FGXXRTK4NBW5GVJNFRPQOCQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.548646410 (518h25m9.419729455s) [noise=true , address=145.79.227.2:4115    , piece_id=MEJXFFNVYOWW27KXKKO5LVUAHADRDZ5AO6YBHSEICTYYIE6WU4OA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.549949401 (518h25m9.421032436s) [noise=true , address=145.79.227.49:4337   , piece_id=7TRMCRIOYQMPHIT3S23ZAHSZA2KQDFYDUCU2RBNBIHZNGWAKCVCQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:11.161782694 (518h25m11.032865729s) [upload=454841]: Next returned nil segment
Dec 27 05:54:11.161793294 (518h25m11.032876339s) [upload=454841]: Waiting for error group managing segments...
Dec 27 05:54:11.548896634 (518h25m11.419979679s) [noise=true , address=145.79.227.4:4185    , piece_id=Q4S6EOET5WVPB3YOIFAYKJFMDD2MXZHIK7FBYH2V3LOQEYLKCMTQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.549696718 (518h25m11.420779753s) [noise=true , address=145.79.227.6:4257    , piece_id=7QDSGX6NHIOIQL2MTABI3FWZSPN4UVROOVTOPZOXVURS4KEP5DSA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.550576012 (518h25m11.421659057s) [noise=true , address=145.79.227.43:4158   , piece_id=JLSZMH5VGAJRQIBR6N4C2LIBRFVAVTB6IU5UUTIW6RN3DH66OWSA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.551044134 (518h25m11.422127179s) [noise=true , address=145.79.227.49:4337   , piece_id=7TRMCRIOYQMPHIT3S23ZAHSZA2KQDFYDUCU2RBNBIHZNGWAKCVCQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.555352657 (518h25m11.426435692s) [noise=true , address=145.79.227.26:4654   , piece_id=33Q6U3MJS3UUWPCGKUD62BL4GLB6WLYXJQQBR5QI3WCNFL3MOTOA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.559356813 (518h25m11.430439858s) [noise=true , address=145.79.227.48:4301   , piece_id=SPCBYJH2254DRQATPEVP5KR7DOM4CIDA7N5Y5CKRYOA2QAZAYDVA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.573596597 (518h25m11.444679643s) [noise=true , address=145.79.227.38:4979   , piece_id=D67D5ENRB5DTU2NYU6VLYQMM763TLYOUCVD5K3IWB3U4UQLG47HA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.574172554 (518h25m11.445255599s) [noise=true , address=145.79.227.2:4115    , piece_id=MEJXFFNVYOWW27KXKKO5LVUAHADRDZ5AO6YBHSEICTYYIE6WU4OA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.579111118 (518h25m11.450194163s) [noise=true , address=145.79.227.9:4366    , piece_id=Q64OQVUCVW7F6F4ZLU7VNXG4BPWLDJGFLIMNG7MWECGSZ3D7QW5Q, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.580063070 (518h25m11.451146115s) [noise=true , address=145.79.227.45:4229   , piece_id=4LMM6JU7AZLTYY5DE7DISC6CF3TNMV57QCBVKKDMZW477PVP3NUQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.586337628 (518h25m11.457420663s) [noise=true , address=145.79.227.42:4121   , piece_id=KVCHRVKHNP5C3ZGZN5SQLZ7PYFCWS433K2S2EWYDT42TWPA5D6KQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.799600647 (518h25m11.670683692s) [noise=true , address=45.91.116.11:4118    , piece_id=DNJN7GRTCG5FNW4U7D5MV3TFZMSOUEJCBVZI7NJ6K3ZKLCWGKWVQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.816634772 (518h25m11.687717817s) [noise=true , address=45.91.116.13:4155    , piece_id=BAU2227SSFDURRDCFQKENLCS3DW7VGGOMOL5S2SMWYEBQ6FA6E7A, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.816658448 (518h25m11.687741493s) [seg_pos={0 0}, upload=454841]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:54:11.816886862 (518h25m11.687969907s) [noise=true , address=45.91.116.4:4939     , piece_id=FRTWR72DXMVKTEKPWSJALHLDAZUZ3FGXXRTK4NBW5GVJNFRPQOCQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 05:54:11.816894738 (518h25m11.687977773s) [noise=true , address=45.91.116.4:4939     , piece_id=FRTWR72DXMVKTEKPWSJALHLDAZUZ3FGXXRTK4NBW5GVJNFRPQOCQ, seg_pos={0 0}, upload=454841]: Long tail cancellation due to optimal threshold
Dec 27 05:54:11.816936808 (518h25m11.688019853s) [noise=true , address=45.91.116.12:4226    , piece_id=HAUZMVQJ5PVFSHJYB6B623MRPEDFXPAPOOIEHGGWS22RSZFTMURQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 27 05:54:11.816946867 (518h25m11.688029902s) [noise=true , address=45.91.116.12:4226    , piece_id=HAUZMVQJ5PVFSHJYB6B623MRPEDFXPAPOOIEHGGWS22RSZFTMURQ, seg_pos={0 0}, upload=454841]: Long tail cancellation due to optimal threshold
Dec 27 05:54:11.817107924 (518h25m11.688190969s) [noise=true , address=77.90.31.10:4520     , piece_id=BBE2ZFKCBKJUU4O7CKQM7EMGUMPAXF3RJJZVY4BZK3XD2LBQM5QA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 27 05:54:11.817116279 (518h25m11.688199314s) [noise=true , address=77.90.31.10:4520     , piece_id=BBE2ZFKCBKJUU4O7CKQM7EMGUMPAXF3RJJZVY4BZK3XD2LBQM5QA, seg_pos={0 0}, upload=454841]: Long tail cancellation due to optimal threshold
Dec 27 05:54:11.817139424 (518h25m11.688222459s) [seg_pos={0 0}, upload=454841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:54:11.817229314 (518h25m11.68831236s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:54:11.817235938 (518h25m11.688318983s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:54:12.112139664 (518h25m11.983222699s) [upload=454841]: Done uploading segments...
Dec 27 05:54:12.112159773 (518h25m11.983242808s) [upload=454841]: Upload finished. err: <nil>
Duration: 5.746969151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..................X.............X...........................]
duration: 
41.061553951s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:55:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 41.061573959s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...............................................X............]
duration: 
3.645756848s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:55:16 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.645785032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [...XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X.....X.]
duration: 
6.125459321s
trace: 
https://jaeger.dev.storj.io/trace/6e98bce886072abb
FAIL (2025-12-27 05:55:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:54:09.292640120 (518h25m9.163723175s) [upload=454841]: Starting upload
Dec 27 05:54:09.292721946 (518h25m9.163804992s) [upload=454841]: Done starting upload
Dec 27 05:54:09.292926666 (518h25m9.164009701s) [upload=454841]: Uploading segments...
Dec 27 05:54:09.292961653 (518h25m9.164044698s) [upload=454841]: Got next segment. Inline: false
Dec 27 05:54:09.292984676 (518h25m9.164067711s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:54:09.292991269 (518h25m9.164074304s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:54:09.541202385 (518h25m9.41228542s) [seg_pos={0 0}, upload=454841]: Begin upload segment...
Dec 27 05:54:09.541255896 (518h25m9.412338942s) [seg_pos={0 0}, upload=454841]: Done begin upload segment.
Dec 27 05:54:09.541280934 (518h25m9.412363969s) [noise=true , address=145.79.227.4:4185    , piece_id=Q4S6EOET5WVPB3YOIFAYKJFMDD2MXZHIK7FBYH2V3LOQEYLKCMTQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.541589151 (518h25m9.412672197s) [noise=true , address=145.79.227.38:4979   , piece_id=D67D5ENRB5DTU2NYU6VLYQMM763TLYOUCVD5K3IWB3U4UQLG47HA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.541717636 (518h25m9.412800682s) [noise=true , address=145.79.227.45:4229   , piece_id=4LMM6JU7AZLTYY5DE7DISC6CF3TNMV57QCBVKKDMZW477PVP3NUQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.542275909 (518h25m9.413358945s) [noise=true , address=145.79.227.6:4257    , piece_id=7QDSGX6NHIOIQL2MTABI3FWZSPN4UVROOVTOPZOXVURS4KEP5DSA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.542616096 (518h25m9.413699132s) [noise=true , address=77.90.31.10:4520     , piece_id=BBE2ZFKCBKJUU4O7CKQM7EMGUMPAXF3RJJZVY4BZK3XD2LBQM5QA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.542940395 (518h25m9.41402343s) [noise=true , address=145.79.227.43:4158   , piece_id=JLSZMH5VGAJRQIBR6N4C2LIBRFVAVTB6IU5UUTIW6RN3DH66OWSA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.543213785 (518h25m9.41429683s) [noise=true , address=45.91.116.12:4226    , piece_id=HAUZMVQJ5PVFSHJYB6B623MRPEDFXPAPOOIEHGGWS22RSZFTMURQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.543528014 (518h25m9.414611059s) [noise=true , address=145.79.227.9:4366    , piece_id=Q64OQVUCVW7F6F4ZLU7VNXG4BPWLDJGFLIMNG7MWECGSZ3D7QW5Q, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.543894992 (518h25m9.414978037s) [noise=true , address=145.79.227.48:4301   , piece_id=SPCBYJH2254DRQATPEVP5KR7DOM4CIDA7N5Y5CKRYOA2QAZAYDVA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.544295034 (518h25m9.415378069s) [noise=true , address=45.91.116.11:4118    , piece_id=DNJN7GRTCG5FNW4U7D5MV3TFZMSOUEJCBVZI7NJ6K3ZKLCWGKWVQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.544661251 (518h25m9.415744286s) [noise=true , address=145.79.227.42:4121   , piece_id=KVCHRVKHNP5C3ZGZN5SQLZ7PYFCWS433K2S2EWYDT42TWPA5D6KQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.545507773 (518h25m9.416590808s) [noise=true , address=145.79.227.26:4654   , piece_id=33Q6U3MJS3UUWPCGKUD62BL4GLB6WLYXJQQBR5QI3WCNFL3MOTOA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.546101915 (518h25m9.41718495s) [noise=true , address=45.91.116.13:4155    , piece_id=BAU2227SSFDURRDCFQKENLCS3DW7VGGOMOL5S2SMWYEBQ6FA6E7A, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.547363156 (518h25m9.418446191s) [noise=true , address=45.91.116.4:4939     , piece_id=FRTWR72DXMVKTEKPWSJALHLDAZUZ3FGXXRTK4NBW5GVJNFRPQOCQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.548646410 (518h25m9.419729455s) [noise=true , address=145.79.227.2:4115    , piece_id=MEJXFFNVYOWW27KXKKO5LVUAHADRDZ5AO6YBHSEICTYYIE6WU4OA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.549949401 (518h25m9.421032436s) [noise=true , address=145.79.227.49:4337   , piece_id=7TRMCRIOYQMPHIT3S23ZAHSZA2KQDFYDUCU2RBNBIHZNGWAKCVCQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:11.161782694 (518h25m11.032865729s) [upload=454841]: Next returned nil segment
Dec 27 05:54:11.161793294 (518h25m11.032876339s) [upload=454841]: Waiting for error group managing segments...
Dec 27 05:54:11.548896634 (518h25m11.419979679s) [noise=true , address=145.79.227.4:4185    , piece_id=Q4S6EOET5WVPB3YOIFAYKJFMDD2MXZHIK7FBYH2V3LOQEYLKCMTQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.549696718 (518h25m11.420779753s) [noise=true , address=145.79.227.6:4257    , piece_id=7QDSGX6NHIOIQL2MTABI3FWZSPN4UVROOVTOPZOXVURS4KEP5DSA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.550576012 (518h25m11.421659057s) [noise=true , address=145.79.227.43:4158   , piece_id=JLSZMH5VGAJRQIBR6N4C2LIBRFVAVTB6IU5UUTIW6RN3DH66OWSA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.551044134 (518h25m11.422127179s) [noise=true , address=145.79.227.49:4337   , piece_id=7TRMCRIOYQMPHIT3S23ZAHSZA2KQDFYDUCU2RBNBIHZNGWAKCVCQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.555352657 (518h25m11.426435692s) [noise=true , address=145.79.227.26:4654   , piece_id=33Q6U3MJS3UUWPCGKUD62BL4GLB6WLYXJQQBR5QI3WCNFL3MOTOA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.559356813 (518h25m11.430439858s) [noise=true , address=145.79.227.48:4301   , piece_id=SPCBYJH2254DRQATPEVP5KR7DOM4CIDA7N5Y5CKRYOA2QAZAYDVA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.573596597 (518h25m11.444679643s) [noise=true , address=145.79.227.38:4979   , piece_id=D67D5ENRB5DTU2NYU6VLYQMM763TLYOUCVD5K3IWB3U4UQLG47HA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.574172554 (518h25m11.445255599s) [noise=true , address=145.79.227.2:4115    , piece_id=MEJXFFNVYOWW27KXKKO5LVUAHADRDZ5AO6YBHSEICTYYIE6WU4OA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.579111118 (518h25m11.450194163s) [noise=true , address=145.79.227.9:4366    , piece_id=Q64OQVUCVW7F6F4ZLU7VNXG4BPWLDJGFLIMNG7MWECGSZ3D7QW5Q, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.580063070 (518h25m11.451146115s) [noise=true , address=145.79.227.45:4229   , piece_id=4LMM6JU7AZLTYY5DE7DISC6CF3TNMV57QCBVKKDMZW477PVP3NUQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.586337628 (518h25m11.457420663s) [noise=true , address=145.79.227.42:4121   , piece_id=KVCHRVKHNP5C3ZGZN5SQLZ7PYFCWS433K2S2EWYDT42TWPA5D6KQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.799600647 (518h25m11.670683692s) [noise=true , address=45.91.116.11:4118    , piece_id=DNJN7GRTCG5FNW4U7D5MV3TFZMSOUEJCBVZI7NJ6K3ZKLCWGKWVQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.816634772 (518h25m11.687717817s) [noise=true , address=45.91.116.13:4155    , piece_id=BAU2227SSFDURRDCFQKENLCS3DW7VGGOMOL5S2SMWYEBQ6FA6E7A, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.816658448 (518h25m11.687741493s) [seg_pos={0 0}, upload=454841]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:54:11.816886862 (518h25m11.687969907s) [noise=true , address=45.91.116.4:4939     , piece_id=FRTWR72DXMVKTEKPWSJALHLDAZUZ3FGXXRTK4NBW5GVJNFRPQOCQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 05:54:11.816894738 (518h25m11.687977773s) [noise=true , address=45.91.116.4:4939     , piece_id=FRTWR72DXMVKTEKPWSJALHLDAZUZ3FGXXRTK4NBW5GVJNFRPQOCQ, seg_pos={0 0}, upload=454841]: Long tail cancellation due to optimal threshold
Dec 27 05:54:11.816936808 (518h25m11.688019853s) [noise=true , address=45.91.116.12:4226    , piece_id=HAUZMVQJ5PVFSHJYB6B623MRPEDFXPAPOOIEHGGWS22RSZFTMURQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 27 05:54:11.816946867 (518h25m11.688029902s) [noise=true , address=45.91.116.12:4226    , piece_id=HAUZMVQJ5PVFSHJYB6B623MRPEDFXPAPOOIEHGGWS22RSZFTMURQ, seg_pos={0 0}, upload=454841]: Long tail cancellation due to optimal threshold
Dec 27 05:54:11.817107924 (518h25m11.688190969s) [noise=true , address=77.90.31.10:4520     , piece_id=BBE2ZFKCBKJUU4O7CKQM7EMGUMPAXF3RJJZVY4BZK3XD2LBQM5QA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 27 05:54:11.817116279 (518h25m11.688199314s) [noise=true , address=77.90.31.10:4520     , piece_id=BBE2ZFKCBKJUU4O7CKQM7EMGUMPAXF3RJJZVY4BZK3XD2LBQM5QA, seg_pos={0 0}, upload=454841]: Long tail cancellation due to optimal threshold
Dec 27 05:54:11.817139424 (518h25m11.688222459s) [seg_pos={0 0}, upload=454841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:54:11.817229314 (518h25m11.68831236s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:54:11.817235938 (518h25m11.688318983s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:54:12.112139664 (518h25m11.983222699s) [upload=454841]: Done uploading segments...
Dec 27 05:54:12.112159773 (518h25m11.983242808s) [upload=454841]: Upload finished. err: <nil>
Duration: 6.125491763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [....XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X.....X]
duration: 
30.002373879s
trace: 
https://jaeger.dev.storj.io/trace/6e98bce886072abb
FAIL (2025-12-27 05:54:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:54:09.292640120 (518h25m9.163723175s) [upload=454841]: Starting upload
Dec 27 05:54:09.292721946 (518h25m9.163804992s) [upload=454841]: Done starting upload
Dec 27 05:54:09.292926666 (518h25m9.164009701s) [upload=454841]: Uploading segments...
Dec 27 05:54:09.292961653 (518h25m9.164044698s) [upload=454841]: Got next segment. Inline: false
Dec 27 05:54:09.292984676 (518h25m9.164067711s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:54:09.292991269 (518h25m9.164074304s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:54:09.541202385 (518h25m9.41228542s) [seg_pos={0 0}, upload=454841]: Begin upload segment...
Dec 27 05:54:09.541255896 (518h25m9.412338942s) [seg_pos={0 0}, upload=454841]: Done begin upload segment.
Dec 27 05:54:09.541280934 (518h25m9.412363969s) [noise=true , address=145.79.227.4:4185    , piece_id=Q4S6EOET5WVPB3YOIFAYKJFMDD2MXZHIK7FBYH2V3LOQEYLKCMTQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.541589151 (518h25m9.412672197s) [noise=true , address=145.79.227.38:4979   , piece_id=D67D5ENRB5DTU2NYU6VLYQMM763TLYOUCVD5K3IWB3U4UQLG47HA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.541717636 (518h25m9.412800682s) [noise=true , address=145.79.227.45:4229   , piece_id=4LMM6JU7AZLTYY5DE7DISC6CF3TNMV57QCBVKKDMZW477PVP3NUQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.542275909 (518h25m9.413358945s) [noise=true , address=145.79.227.6:4257    , piece_id=7QDSGX6NHIOIQL2MTABI3FWZSPN4UVROOVTOPZOXVURS4KEP5DSA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.542616096 (518h25m9.413699132s) [noise=true , address=77.90.31.10:4520     , piece_id=BBE2ZFKCBKJUU4O7CKQM7EMGUMPAXF3RJJZVY4BZK3XD2LBQM5QA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.542940395 (518h25m9.41402343s) [noise=true , address=145.79.227.43:4158   , piece_id=JLSZMH5VGAJRQIBR6N4C2LIBRFVAVTB6IU5UUTIW6RN3DH66OWSA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.543213785 (518h25m9.41429683s) [noise=true , address=45.91.116.12:4226    , piece_id=HAUZMVQJ5PVFSHJYB6B623MRPEDFXPAPOOIEHGGWS22RSZFTMURQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.543528014 (518h25m9.414611059s) [noise=true , address=145.79.227.9:4366    , piece_id=Q64OQVUCVW7F6F4ZLU7VNXG4BPWLDJGFLIMNG7MWECGSZ3D7QW5Q, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.543894992 (518h25m9.414978037s) [noise=true , address=145.79.227.48:4301   , piece_id=SPCBYJH2254DRQATPEVP5KR7DOM4CIDA7N5Y5CKRYOA2QAZAYDVA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.544295034 (518h25m9.415378069s) [noise=true , address=45.91.116.11:4118    , piece_id=DNJN7GRTCG5FNW4U7D5MV3TFZMSOUEJCBVZI7NJ6K3ZKLCWGKWVQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.544661251 (518h25m9.415744286s) [noise=true , address=145.79.227.42:4121   , piece_id=KVCHRVKHNP5C3ZGZN5SQLZ7PYFCWS433K2S2EWYDT42TWPA5D6KQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.545507773 (518h25m9.416590808s) [noise=true , address=145.79.227.26:4654   , piece_id=33Q6U3MJS3UUWPCGKUD62BL4GLB6WLYXJQQBR5QI3WCNFL3MOTOA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.546101915 (518h25m9.41718495s) [noise=true , address=45.91.116.13:4155    , piece_id=BAU2227SSFDURRDCFQKENLCS3DW7VGGOMOL5S2SMWYEBQ6FA6E7A, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.547363156 (518h25m9.418446191s) [noise=true , address=45.91.116.4:4939     , piece_id=FRTWR72DXMVKTEKPWSJALHLDAZUZ3FGXXRTK4NBW5GVJNFRPQOCQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.548646410 (518h25m9.419729455s) [noise=true , address=145.79.227.2:4115    , piece_id=MEJXFFNVYOWW27KXKKO5LVUAHADRDZ5AO6YBHSEICTYYIE6WU4OA, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:09.549949401 (518h25m9.421032436s) [noise=true , address=145.79.227.49:4337   , piece_id=7TRMCRIOYQMPHIT3S23ZAHSZA2KQDFYDUCU2RBNBIHZNGWAKCVCQ, seg_pos={0 0}, upload=454841]: Uploading piece...
Dec 27 05:54:11.161782694 (518h25m11.032865729s) [upload=454841]: Next returned nil segment
Dec 27 05:54:11.161793294 (518h25m11.032876339s) [upload=454841]: Waiting for error group managing segments...
Dec 27 05:54:11.548896634 (518h25m11.419979679s) [noise=true , address=145.79.227.4:4185    , piece_id=Q4S6EOET5WVPB3YOIFAYKJFMDD2MXZHIK7FBYH2V3LOQEYLKCMTQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.549696718 (518h25m11.420779753s) [noise=true , address=145.79.227.6:4257    , piece_id=7QDSGX6NHIOIQL2MTABI3FWZSPN4UVROOVTOPZOXVURS4KEP5DSA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.550576012 (518h25m11.421659057s) [noise=true , address=145.79.227.43:4158   , piece_id=JLSZMH5VGAJRQIBR6N4C2LIBRFVAVTB6IU5UUTIW6RN3DH66OWSA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.551044134 (518h25m11.422127179s) [noise=true , address=145.79.227.49:4337   , piece_id=7TRMCRIOYQMPHIT3S23ZAHSZA2KQDFYDUCU2RBNBIHZNGWAKCVCQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.555352657 (518h25m11.426435692s) [noise=true , address=145.79.227.26:4654   , piece_id=33Q6U3MJS3UUWPCGKUD62BL4GLB6WLYXJQQBR5QI3WCNFL3MOTOA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.559356813 (518h25m11.430439858s) [noise=true , address=145.79.227.48:4301   , piece_id=SPCBYJH2254DRQATPEVP5KR7DOM4CIDA7N5Y5CKRYOA2QAZAYDVA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.573596597 (518h25m11.444679643s) [noise=true , address=145.79.227.38:4979   , piece_id=D67D5ENRB5DTU2NYU6VLYQMM763TLYOUCVD5K3IWB3U4UQLG47HA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.574172554 (518h25m11.445255599s) [noise=true , address=145.79.227.2:4115    , piece_id=MEJXFFNVYOWW27KXKKO5LVUAHADRDZ5AO6YBHSEICTYYIE6WU4OA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.579111118 (518h25m11.450194163s) [noise=true , address=145.79.227.9:4366    , piece_id=Q64OQVUCVW7F6F4ZLU7VNXG4BPWLDJGFLIMNG7MWECGSZ3D7QW5Q, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.580063070 (518h25m11.451146115s) [noise=true , address=145.79.227.45:4229   , piece_id=4LMM6JU7AZLTYY5DE7DISC6CF3TNMV57QCBVKKDMZW477PVP3NUQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.586337628 (518h25m11.457420663s) [noise=true , address=145.79.227.42:4121   , piece_id=KVCHRVKHNP5C3ZGZN5SQLZ7PYFCWS433K2S2EWYDT42TWPA5D6KQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.799600647 (518h25m11.670683692s) [noise=true , address=45.91.116.11:4118    , piece_id=DNJN7GRTCG5FNW4U7D5MV3TFZMSOUEJCBVZI7NJ6K3ZKLCWGKWVQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.816634772 (518h25m11.687717817s) [noise=true , address=45.91.116.13:4155    , piece_id=BAU2227SSFDURRDCFQKENLCS3DW7VGGOMOL5S2SMWYEBQ6FA6E7A, seg_pos={0 0}, upload=454841]: Done uploading piece. err: <nil>
Dec 27 05:54:11.816658448 (518h25m11.687741493s) [seg_pos={0 0}, upload=454841]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:54:11.816886862 (518h25m11.687969907s) [noise=true , address=45.91.116.4:4939     , piece_id=FRTWR72DXMVKTEKPWSJALHLDAZUZ3FGXXRTK4NBW5GVJNFRPQOCQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 05:54:11.816894738 (518h25m11.687977773s) [noise=true , address=45.91.116.4:4939     , piece_id=FRTWR72DXMVKTEKPWSJALHLDAZUZ3FGXXRTK4NBW5GVJNFRPQOCQ, seg_pos={0 0}, upload=454841]: Long tail cancellation due to optimal threshold
Dec 27 05:54:11.816936808 (518h25m11.688019853s) [noise=true , address=45.91.116.12:4226    , piece_id=HAUZMVQJ5PVFSHJYB6B623MRPEDFXPAPOOIEHGGWS22RSZFTMURQ, seg_pos={0 0}, upload=454841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 27 05:54:11.816946867 (518h25m11.688029902s) [noise=true , address=45.91.116.12:4226    , piece_id=HAUZMVQJ5PVFSHJYB6B623MRPEDFXPAPOOIEHGGWS22RSZFTMURQ, seg_pos={0 0}, upload=454841]: Long tail cancellation due to optimal threshold
Dec 27 05:54:11.817107924 (518h25m11.688190969s) [noise=true , address=77.90.31.10:4520     , piece_id=BBE2ZFKCBKJUU4O7CKQM7EMGUMPAXF3RJJZVY4BZK3XD2LBQM5QA, seg_pos={0 0}, upload=454841]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 27 05:54:11.817116279 (518h25m11.688199314s) [noise=true , address=77.90.31.10:4520     , piece_id=BBE2ZFKCBKJUU4O7CKQM7EMGUMPAXF3RJJZVY4BZK3XD2LBQM5QA, seg_pos={0 0}, upload=454841]: Long tail cancellation due to optimal threshold
Dec 27 05:54:11.817139424 (518h25m11.688222459s) [seg_pos={0 0}, upload=454841]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:54:11.817229314 (518h25m11.68831236s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:54:11.817235938 (518h25m11.688318983s) [upload=454841]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:54:12.112139664 (518h25m11.983222699s) [upload=454841]: Done uploading segments...
Dec 27 05:54:12.112159773 (518h25m11.983242808s) [upload=454841]: Upload finished. err: <nil>
Duration: 30.002387254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...................X.............X..........................]
duration: 
42.66509521s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:54:19 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 42.665125899s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [X...............................................X...........]
duration: 
4.25183955s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:54:14 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.251856201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [.X...............................................X..........]
duration: 
3.545568893s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:53:50 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.545586226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [....................X.............X.........................]
duration: 
47.190322322s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:53:19 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 47.19034253s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [..X...............................................X.........]
duration: 
3.516263962s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:53:08 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.516296756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.....XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X.....]
duration: 
5.679226507s
trace: 
https://jaeger.dev.storj.io/trace/43b2f940a7e57180
FAIL (2025-12-27 05:52:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:50:29.633789985 (518h21m29.50487302s) [upload=454781]: Starting upload
Dec 27 05:50:29.633920223 (518h21m29.505003268s) [upload=454781]: Done starting upload
Dec 27 05:50:29.634019502 (518h21m29.505102537s) [upload=454781]: Uploading segments...
Dec 27 05:50:29.634199555 (518h21m29.50528259s) [upload=454781]: Got next segment. Inline: false
Dec 27 05:50:29.634242146 (518h21m29.505325191s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:50:29.634248078 (518h21m29.505331123s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:50:29.874147036 (518h21m29.745230071s) [seg_pos={0 0}, upload=454781]: Begin upload segment...
Dec 27 05:50:29.874227760 (518h21m29.745310805s) [seg_pos={0 0}, upload=454781]: Done begin upload segment.
Dec 27 05:50:29.874248789 (518h21m29.745331824s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.874328311 (518h21m29.745411347s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875025659 (518h21m29.746108704s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875374644 (518h21m29.746457679s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875698691 (518h21m29.746781736s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876041473 (518h21m29.747124508s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876441986 (518h21m29.747525021s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876806109 (518h21m29.747889145s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.877371947 (518h21m29.748454982s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.878910576 (518h21m29.749993621s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.880565368 (518h21m29.751648413s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.882140146 (518h21m29.753223191s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.883876212 (518h21m29.754959257s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.886634314 (518h21m29.757717359s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.889994773 (518h21m29.761077818s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.890629481 (518h21m29.761712517s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:31.517489892 (518h21m31.388572927s) [upload=454781]: Next returned nil segment
Dec 27 05:50:31.517544185 (518h21m31.38862722s) [upload=454781]: Waiting for error group managing segments...
Dec 27 05:50:31.904914325 (518h21m31.77599737s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.905458440 (518h21m31.776541475s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.906393881 (518h21m31.777476927s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.908610543 (518h21m31.779693578s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.916696710 (518h21m31.787779755s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.918054545 (518h21m31.78913758s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.919659241 (518h21m31.790742286s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923116615 (518h21m31.79419966s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923733058 (518h21m31.794816093s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.924225967 (518h21m31.795309012s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.927629478 (518h21m31.798712513s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.168095527 (518h21m32.039178572s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172414721 (518h21m32.043497766s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172487089 (518h21m32.043570134s) [seg_pos={0 0}, upload=454781]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:50:32.172900326 (518h21m32.043983371s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection
Dec 27 05:50:32.172910625 (518h21m32.04399367s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173364900 (518h21m32.044447945s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173374919 (518h21m32.044457954s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173635896 (518h21m32.044718941s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173645755 (518h21m32.0447288s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173671204 (518h21m32.044754239s) [seg_pos={0 0}, upload=454781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:50:32.173949564 (518h21m32.045032609s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:50:32.173959202 (518h21m32.045042237s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:50:32.461346005 (518h21m32.33242904s) [upload=454781]: Done uploading segments...
Dec 27 05:50:32.461366444 (518h21m32.33244949s) [upload=454781]: Upload finished. err: <nil>
Duration: 5.679272154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.....................X.............X........................]
duration: 
35.9678195s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:52:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 35.96784641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X....]
duration: 
5.989336468s
trace: 
https://jaeger.dev.storj.io/trace/43b2f940a7e57180
FAIL (2025-12-27 05:52:27 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:50:29.633789985 (518h21m29.50487302s) [upload=454781]: Starting upload
Dec 27 05:50:29.633920223 (518h21m29.505003268s) [upload=454781]: Done starting upload
Dec 27 05:50:29.634019502 (518h21m29.505102537s) [upload=454781]: Uploading segments...
Dec 27 05:50:29.634199555 (518h21m29.50528259s) [upload=454781]: Got next segment. Inline: false
Dec 27 05:50:29.634242146 (518h21m29.505325191s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:50:29.634248078 (518h21m29.505331123s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:50:29.874147036 (518h21m29.745230071s) [seg_pos={0 0}, upload=454781]: Begin upload segment...
Dec 27 05:50:29.874227760 (518h21m29.745310805s) [seg_pos={0 0}, upload=454781]: Done begin upload segment.
Dec 27 05:50:29.874248789 (518h21m29.745331824s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.874328311 (518h21m29.745411347s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875025659 (518h21m29.746108704s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875374644 (518h21m29.746457679s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875698691 (518h21m29.746781736s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876041473 (518h21m29.747124508s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876441986 (518h21m29.747525021s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876806109 (518h21m29.747889145s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.877371947 (518h21m29.748454982s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.878910576 (518h21m29.749993621s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.880565368 (518h21m29.751648413s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.882140146 (518h21m29.753223191s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.883876212 (518h21m29.754959257s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.886634314 (518h21m29.757717359s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.889994773 (518h21m29.761077818s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.890629481 (518h21m29.761712517s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:31.517489892 (518h21m31.388572927s) [upload=454781]: Next returned nil segment
Dec 27 05:50:31.517544185 (518h21m31.38862722s) [upload=454781]: Waiting for error group managing segments...
Dec 27 05:50:31.904914325 (518h21m31.77599737s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.905458440 (518h21m31.776541475s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.906393881 (518h21m31.777476927s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.908610543 (518h21m31.779693578s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.916696710 (518h21m31.787779755s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.918054545 (518h21m31.78913758s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.919659241 (518h21m31.790742286s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923116615 (518h21m31.79419966s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923733058 (518h21m31.794816093s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.924225967 (518h21m31.795309012s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.927629478 (518h21m31.798712513s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.168095527 (518h21m32.039178572s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172414721 (518h21m32.043497766s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172487089 (518h21m32.043570134s) [seg_pos={0 0}, upload=454781]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:50:32.172900326 (518h21m32.043983371s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection
Dec 27 05:50:32.172910625 (518h21m32.04399367s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173364900 (518h21m32.044447945s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173374919 (518h21m32.044457954s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173635896 (518h21m32.044718941s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173645755 (518h21m32.0447288s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173671204 (518h21m32.044754239s) [seg_pos={0 0}, upload=454781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:50:32.173949564 (518h21m32.045032609s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:50:32.173959202 (518h21m32.045042237s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:50:32.461346005 (518h21m32.33242904s) [upload=454781]: Done uploading segments...
Dec 27 05:50:32.461366444 (518h21m32.33244949s) [upload=454781]: Upload finished. err: <nil>
Duration: 5.989355295s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [...X...............................................X........]
duration: 
3.935168277s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:52:10 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.935194326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X...]
duration: 
14.173335927s
trace: 
https://jaeger.dev.storj.io/trace/43b2f940a7e57180
FAIL (2025-12-27 05:52:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:50:29.633789985 (518h21m29.50487302s) [upload=454781]: Starting upload
Dec 27 05:50:29.633920223 (518h21m29.505003268s) [upload=454781]: Done starting upload
Dec 27 05:50:29.634019502 (518h21m29.505102537s) [upload=454781]: Uploading segments...
Dec 27 05:50:29.634199555 (518h21m29.50528259s) [upload=454781]: Got next segment. Inline: false
Dec 27 05:50:29.634242146 (518h21m29.505325191s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:50:29.634248078 (518h21m29.505331123s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:50:29.874147036 (518h21m29.745230071s) [seg_pos={0 0}, upload=454781]: Begin upload segment...
Dec 27 05:50:29.874227760 (518h21m29.745310805s) [seg_pos={0 0}, upload=454781]: Done begin upload segment.
Dec 27 05:50:29.874248789 (518h21m29.745331824s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.874328311 (518h21m29.745411347s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875025659 (518h21m29.746108704s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875374644 (518h21m29.746457679s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875698691 (518h21m29.746781736s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876041473 (518h21m29.747124508s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876441986 (518h21m29.747525021s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876806109 (518h21m29.747889145s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.877371947 (518h21m29.748454982s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.878910576 (518h21m29.749993621s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.880565368 (518h21m29.751648413s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.882140146 (518h21m29.753223191s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.883876212 (518h21m29.754959257s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.886634314 (518h21m29.757717359s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.889994773 (518h21m29.761077818s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.890629481 (518h21m29.761712517s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:31.517489892 (518h21m31.388572927s) [upload=454781]: Next returned nil segment
Dec 27 05:50:31.517544185 (518h21m31.38862722s) [upload=454781]: Waiting for error group managing segments...
Dec 27 05:50:31.904914325 (518h21m31.77599737s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.905458440 (518h21m31.776541475s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.906393881 (518h21m31.777476927s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.908610543 (518h21m31.779693578s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.916696710 (518h21m31.787779755s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.918054545 (518h21m31.78913758s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.919659241 (518h21m31.790742286s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923116615 (518h21m31.79419966s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923733058 (518h21m31.794816093s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.924225967 (518h21m31.795309012s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.927629478 (518h21m31.798712513s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.168095527 (518h21m32.039178572s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172414721 (518h21m32.043497766s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172487089 (518h21m32.043570134s) [seg_pos={0 0}, upload=454781]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:50:32.172900326 (518h21m32.043983371s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection
Dec 27 05:50:32.172910625 (518h21m32.04399367s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173364900 (518h21m32.044447945s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173374919 (518h21m32.044457954s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173635896 (518h21m32.044718941s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173645755 (518h21m32.0447288s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173671204 (518h21m32.044754239s) [seg_pos={0 0}, upload=454781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:50:32.173949564 (518h21m32.045032609s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:50:32.173959202 (518h21m32.045042237s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:50:32.461346005 (518h21m32.33242904s) [upload=454781]: Done uploading segments...
Dec 27 05:50:32.461366444 (518h21m32.33244949s) [upload=454781]: Upload finished. err: <nil>
Duration: 14.17337439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [......................X.............X.......................]
duration: 
32.778250611s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:51:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 32.778270378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X..]
duration: 
5.778932339s
trace: 
https://jaeger.dev.storj.io/trace/43b2f940a7e57180
FAIL (2025-12-27 05:51:21 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:50:29.633789985 (518h21m29.50487302s) [upload=454781]: Starting upload
Dec 27 05:50:29.633920223 (518h21m29.505003268s) [upload=454781]: Done starting upload
Dec 27 05:50:29.634019502 (518h21m29.505102537s) [upload=454781]: Uploading segments...
Dec 27 05:50:29.634199555 (518h21m29.50528259s) [upload=454781]: Got next segment. Inline: false
Dec 27 05:50:29.634242146 (518h21m29.505325191s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:50:29.634248078 (518h21m29.505331123s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:50:29.874147036 (518h21m29.745230071s) [seg_pos={0 0}, upload=454781]: Begin upload segment...
Dec 27 05:50:29.874227760 (518h21m29.745310805s) [seg_pos={0 0}, upload=454781]: Done begin upload segment.
Dec 27 05:50:29.874248789 (518h21m29.745331824s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.874328311 (518h21m29.745411347s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875025659 (518h21m29.746108704s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875374644 (518h21m29.746457679s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875698691 (518h21m29.746781736s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876041473 (518h21m29.747124508s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876441986 (518h21m29.747525021s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876806109 (518h21m29.747889145s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.877371947 (518h21m29.748454982s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.878910576 (518h21m29.749993621s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.880565368 (518h21m29.751648413s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.882140146 (518h21m29.753223191s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.883876212 (518h21m29.754959257s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.886634314 (518h21m29.757717359s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.889994773 (518h21m29.761077818s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.890629481 (518h21m29.761712517s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:31.517489892 (518h21m31.388572927s) [upload=454781]: Next returned nil segment
Dec 27 05:50:31.517544185 (518h21m31.38862722s) [upload=454781]: Waiting for error group managing segments...
Dec 27 05:50:31.904914325 (518h21m31.77599737s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.905458440 (518h21m31.776541475s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.906393881 (518h21m31.777476927s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.908610543 (518h21m31.779693578s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.916696710 (518h21m31.787779755s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.918054545 (518h21m31.78913758s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.919659241 (518h21m31.790742286s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923116615 (518h21m31.79419966s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923733058 (518h21m31.794816093s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.924225967 (518h21m31.795309012s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.927629478 (518h21m31.798712513s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.168095527 (518h21m32.039178572s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172414721 (518h21m32.043497766s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172487089 (518h21m32.043570134s) [seg_pos={0 0}, upload=454781]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:50:32.172900326 (518h21m32.043983371s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection
Dec 27 05:50:32.172910625 (518h21m32.04399367s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173364900 (518h21m32.044447945s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173374919 (518h21m32.044457954s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173635896 (518h21m32.044718941s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173645755 (518h21m32.0447288s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173671204 (518h21m32.044754239s) [seg_pos={0 0}, upload=454781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:50:32.173949564 (518h21m32.045032609s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:50:32.173959202 (518h21m32.045042237s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:50:32.461346005 (518h21m32.33242904s) [upload=454781]: Done uploading segments...
Dec 27 05:50:32.461366444 (518h21m32.33244949s) [upload=454781]: Upload finished. err: <nil>
Duration: 5.778955323s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [....X...............................................X.......]
duration: 
4.408283124s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:51:06 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.408307582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X.]
duration: 
5.999918199s
trace: 
https://jaeger.dev.storj.io/trace/43b2f940a7e57180
FAIL (2025-12-27 05:51:05 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:50:29.633789985 (518h21m29.50487302s) [upload=454781]: Starting upload
Dec 27 05:50:29.633920223 (518h21m29.505003268s) [upload=454781]: Done starting upload
Dec 27 05:50:29.634019502 (518h21m29.505102537s) [upload=454781]: Uploading segments...
Dec 27 05:50:29.634199555 (518h21m29.50528259s) [upload=454781]: Got next segment. Inline: false
Dec 27 05:50:29.634242146 (518h21m29.505325191s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:50:29.634248078 (518h21m29.505331123s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:50:29.874147036 (518h21m29.745230071s) [seg_pos={0 0}, upload=454781]: Begin upload segment...
Dec 27 05:50:29.874227760 (518h21m29.745310805s) [seg_pos={0 0}, upload=454781]: Done begin upload segment.
Dec 27 05:50:29.874248789 (518h21m29.745331824s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.874328311 (518h21m29.745411347s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875025659 (518h21m29.746108704s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875374644 (518h21m29.746457679s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875698691 (518h21m29.746781736s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876041473 (518h21m29.747124508s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876441986 (518h21m29.747525021s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876806109 (518h21m29.747889145s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.877371947 (518h21m29.748454982s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.878910576 (518h21m29.749993621s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.880565368 (518h21m29.751648413s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.882140146 (518h21m29.753223191s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.883876212 (518h21m29.754959257s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.886634314 (518h21m29.757717359s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.889994773 (518h21m29.761077818s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.890629481 (518h21m29.761712517s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:31.517489892 (518h21m31.388572927s) [upload=454781]: Next returned nil segment
Dec 27 05:50:31.517544185 (518h21m31.38862722s) [upload=454781]: Waiting for error group managing segments...
Dec 27 05:50:31.904914325 (518h21m31.77599737s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.905458440 (518h21m31.776541475s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.906393881 (518h21m31.777476927s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.908610543 (518h21m31.779693578s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.916696710 (518h21m31.787779755s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.918054545 (518h21m31.78913758s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.919659241 (518h21m31.790742286s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923116615 (518h21m31.79419966s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923733058 (518h21m31.794816093s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.924225967 (518h21m31.795309012s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.927629478 (518h21m31.798712513s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.168095527 (518h21m32.039178572s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172414721 (518h21m32.043497766s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172487089 (518h21m32.043570134s) [seg_pos={0 0}, upload=454781]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:50:32.172900326 (518h21m32.043983371s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection
Dec 27 05:50:32.172910625 (518h21m32.04399367s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173364900 (518h21m32.044447945s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173374919 (518h21m32.044457954s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173635896 (518h21m32.044718941s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173645755 (518h21m32.0447288s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173671204 (518h21m32.044754239s) [seg_pos={0 0}, upload=454781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:50:32.173949564 (518h21m32.045032609s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:50:32.173959202 (518h21m32.045042237s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:50:32.461346005 (518h21m32.33242904s) [upload=454781]: Done uploading segments...
Dec 27 05:50:32.461366444 (518h21m32.33244949s) [upload=454781]: Upload finished. err: <nil>
Duration: 5.99993953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [X..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......X]
duration: 
30.003921434s
trace: 
https://jaeger.dev.storj.io/trace/43b2f940a7e57180
FAIL (2025-12-27 05:50:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:50:29.633789985 (518h21m29.50487302s) [upload=454781]: Starting upload
Dec 27 05:50:29.633920223 (518h21m29.505003268s) [upload=454781]: Done starting upload
Dec 27 05:50:29.634019502 (518h21m29.505102537s) [upload=454781]: Uploading segments...
Dec 27 05:50:29.634199555 (518h21m29.50528259s) [upload=454781]: Got next segment. Inline: false
Dec 27 05:50:29.634242146 (518h21m29.505325191s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:50:29.634248078 (518h21m29.505331123s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:50:29.874147036 (518h21m29.745230071s) [seg_pos={0 0}, upload=454781]: Begin upload segment...
Dec 27 05:50:29.874227760 (518h21m29.745310805s) [seg_pos={0 0}, upload=454781]: Done begin upload segment.
Dec 27 05:50:29.874248789 (518h21m29.745331824s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.874328311 (518h21m29.745411347s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875025659 (518h21m29.746108704s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875374644 (518h21m29.746457679s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.875698691 (518h21m29.746781736s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876041473 (518h21m29.747124508s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876441986 (518h21m29.747525021s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.876806109 (518h21m29.747889145s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.877371947 (518h21m29.748454982s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.878910576 (518h21m29.749993621s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.880565368 (518h21m29.751648413s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.882140146 (518h21m29.753223191s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.883876212 (518h21m29.754959257s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.886634314 (518h21m29.757717359s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.889994773 (518h21m29.761077818s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:29.890629481 (518h21m29.761712517s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Uploading piece...
Dec 27 05:50:31.517489892 (518h21m31.388572927s) [upload=454781]: Next returned nil segment
Dec 27 05:50:31.517544185 (518h21m31.38862722s) [upload=454781]: Waiting for error group managing segments...
Dec 27 05:50:31.904914325 (518h21m31.77599737s) [noise=true , address=145.79.227.15:4874   , piece_id=JKNP2QXQLD2K4A6GGUVRYSDXKO6KLLOLK6KKIF5YTNQ5E3PARDOQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.905458440 (518h21m31.776541475s) [noise=true , address=145.79.227.42:4122   , piece_id=RQLRK5ULPUB3KF5O6YOVX2IPDVXBEJF5APUMG5XJAR2GSIAD7JVQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.906393881 (518h21m31.777476927s) [noise=true , address=145.79.227.26:4654   , piece_id=FQRLNKV7MO3STL37RVMXXBLYL3S6SHZAKFJ3G27Z6JTB3ZJHHD3Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.908610543 (518h21m31.779693578s) [noise=true , address=145.79.227.12:4765   , piece_id=VUK6UIEHDK4KEE2WRC7UDJXXWLYRKN5SYHHRATJWOE2FDFZLE47A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.916696710 (518h21m31.787779755s) [noise=true , address=145.79.227.37:4565   , piece_id=FONTMHGE5WYJ7C7CXK6CNAEY3QE5EJDI4J7T55EVJRGXJKJKFPFA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.918054545 (518h21m31.78913758s) [noise=true , address=145.79.227.41:4086   , piece_id=ZAOLRXT2INEKJLYETFATUBZIOVDD3ATXQK3XISAXD737HWIKLBMQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.919659241 (518h21m31.790742286s) [noise=true , address=145.79.227.48:4303   , piece_id=UGVULM5U6GYXHC6EBROKHZZ5TFZHOAZTV2KI3TWV4HEJQUTN552Q, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923116615 (518h21m31.79419966s) [noise=true , address=145.79.227.20:4439   , piece_id=MKU67ROMK5GUTVSWXPOU2TGEHDMZALHH6EZO6WTP3DTMKX6C7K3A, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.923733058 (518h21m31.794816093s) [noise=true , address=145.79.227.19:4079   , piece_id=H3OUEMUC5TIOQTQV3LY7T7VU5DVNPX3F2ZVXTXWARRJKOBU32QJQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.924225967 (518h21m31.795309012s) [noise=true , address=145.79.227.50:4373   , piece_id=YGLTPFYBCOQ42566K5BJKVFHGUZJOUF55RYPV63Z63OI5SO4W2SA, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:31.927629478 (518h21m31.798712513s) [noise=true , address=145.79.227.4:4185    , piece_id=Z2Y4F2IP3KUMTHBQZRUNYLUCG34TCKO3LJWCYIVSOQ2PECCB2XNQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.168095527 (518h21m32.039178572s) [noise=true , address=45.91.116.5:4300     , piece_id=6PHUTBDXYPSBGNKWJEX57A4JPH7I6VEJMGVRRTUCU6BIXY2YBSBQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172414721 (518h21m32.043497766s) [noise=true , address=77.90.31.14:4661     , piece_id=ZKXJX6Q4OXDBNSRQAMMGKPU2Y2L3FXLFKVYNQFPOY4FSURAMCWRQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: <nil>
Dec 27 05:50:32.172487089 (518h21m32.043570134s) [seg_pos={0 0}, upload=454781]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:50:32.172900326 (518h21m32.043983371s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44960->45.91.116.7:4011: use of closed network connection
Dec 27 05:50:32.172910625 (518h21m32.04399367s) [noise=true , address=45.91.116.7:4011     , piece_id=CYYVU7DN73FINMJ6PXZ5DB7EDWRM5BVPFZRB2SJKFPDB7U4MWLTQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173364900 (518h21m32.044447945s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173374919 (518h21m32.044457954s) [noise=true , address=45.91.116.10:4083    , piece_id=S6ZFKACMAI3SBSQ5XEY7ZB5IH3HXLCNTS5CWXPZ2J22ESAJWH3TQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173635896 (518h21m32.044718941s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 27 05:50:32.173645755 (518h21m32.0447288s) [noise=true , address=45.91.116.6:4975     , piece_id=UA7OPOFZIJ3ODWWTYEALB4COYCR4LPZH7J3YBHFFHVKRYKSQMWWQ, seg_pos={0 0}, upload=454781]: Long tail cancellation due to optimal threshold
Dec 27 05:50:32.173671204 (518h21m32.044754239s) [seg_pos={0 0}, upload=454781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:50:32.173949564 (518h21m32.045032609s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:50:32.173959202 (518h21m32.045042237s) [upload=454781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:50:32.461346005 (518h21m32.33242904s) [upload=454781]: Done uploading segments...
Dec 27 05:50:32.461366444 (518h21m32.33244949s) [upload=454781]: Upload finished. err: <nil>
Duration: 30.003941704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.......................X.............X......................]
duration: 
28.102998512s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:50:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 28.103015224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [.....X...............................................X......]
duration: 
4.36514792s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:50:29 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.365188717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [........................X.............X.....................]
duration: 
49.442748781s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:50:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 49.442848621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [XX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X......]
duration: 
5.894538698s
trace: 
https://jaeger.dev.storj.io/trace/e6b9cde4f4af8a7
FAIL (2025-12-27 05:50:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA) context deadline exceeded
======LOGS======
Dec 27 05:46:00.348872485 (518h17m0.21995552s) [upload=454711]: Starting upload
Dec 27 05:46:00.348967666 (518h17m0.220050711s) [upload=454711]: Done starting upload
Dec 27 05:46:00.349066805 (518h17m0.22014984s) [upload=454711]: Uploading segments...
Dec 27 05:46:00.349230336 (518h17m0.220313381s) [upload=454711]: Got next segment. Inline: false
Dec 27 05:46:00.349284158 (518h17m0.220367193s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:46:00.349289890 (518h17m0.220372935s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:46:00.596547460 (518h17m0.467630515s) [seg_pos={0 0}, upload=454711]: Begin upload segment...
Dec 27 05:46:00.596600422 (518h17m0.467683467s) [seg_pos={0 0}, upload=454711]: Done begin upload segment.
Dec 27 05:46:00.596626500 (518h17m0.467709535s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596638704 (518h17m0.467721749s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596833695 (518h17m0.46791675s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597223769 (518h17m0.468306814s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597563425 (518h17m0.46864646s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598011227 (518h17m0.469094272s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598422290 (518h17m0.469505325s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598792486 (518h17m0.469875531s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.599359966 (518h17m0.470443011s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600416808 (518h17m0.471499843s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600796782 (518h17m0.471879827s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.601568361 (518h17m0.472651406s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602256040 (518h17m0.473339075s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602745653 (518h17m0.473828688s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.603827122 (518h17m0.474910167s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.604147301 (518h17m0.475230336s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:02.224895133 (518h17m2.095978178s) [upload=454711]: Next returned nil segment
Dec 27 05:46:02.224920952 (518h17m2.096003997s) [upload=454711]: Waiting for error group managing segments...
Dec 27 05:46:02.611896800 (518h17m2.482979835s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.612665283 (518h17m2.483748328s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.618470057 (518h17m2.489553102s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.627484091 (518h17m2.498567126s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.631205627 (518h17m2.502288663s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.632225459 (518h17m2.503308504s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.635912790 (518h17m2.506995836s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637103788 (518h17m2.508186823s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637619290 (518h17m2.508702335s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.638549071 (518h17m2.509632116s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.852825640 (518h17m2.723908685s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.589407397 (518h17m3.460490442s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660687898 (518h17m3.531770943s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660709108 (518h17m3.531792153s) [seg_pos={0 0}, upload=454711]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:46:03.660958513 (518h17m3.532041548s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.660966819 (518h17m3.532049854s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661122996 (518h17m3.532206041s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661130670 (518h17m3.532213715s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661259536 (518h17m3.532342581s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661266960 (518h17m3.532350005s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661283091 (518h17m3.532366126s) [seg_pos={0 0}, upload=454711]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:46:03.661385787 (518h17m3.532468822s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:46:03.661391798 (518h17m3.532474833s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:46:03.949073754 (518h17m3.820156789s) [upload=454711]: Done uploading segments...
Dec 27 05:46:03.949109782 (518h17m3.820192827s) [upload=454711]: Upload finished. err: <nil>
Duration: 5.894570367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X.....]
duration: 
5.702083542s
trace: 
https://jaeger.dev.storj.io/trace/e6b9cde4f4af8a7
FAIL (2025-12-27 05:49:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA) context deadline exceeded
======LOGS======
Dec 27 05:46:00.348872485 (518h17m0.21995552s) [upload=454711]: Starting upload
Dec 27 05:46:00.348967666 (518h17m0.220050711s) [upload=454711]: Done starting upload
Dec 27 05:46:00.349066805 (518h17m0.22014984s) [upload=454711]: Uploading segments...
Dec 27 05:46:00.349230336 (518h17m0.220313381s) [upload=454711]: Got next segment. Inline: false
Dec 27 05:46:00.349284158 (518h17m0.220367193s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:46:00.349289890 (518h17m0.220372935s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:46:00.596547460 (518h17m0.467630515s) [seg_pos={0 0}, upload=454711]: Begin upload segment...
Dec 27 05:46:00.596600422 (518h17m0.467683467s) [seg_pos={0 0}, upload=454711]: Done begin upload segment.
Dec 27 05:46:00.596626500 (518h17m0.467709535s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596638704 (518h17m0.467721749s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596833695 (518h17m0.46791675s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597223769 (518h17m0.468306814s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597563425 (518h17m0.46864646s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598011227 (518h17m0.469094272s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598422290 (518h17m0.469505325s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598792486 (518h17m0.469875531s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.599359966 (518h17m0.470443011s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600416808 (518h17m0.471499843s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600796782 (518h17m0.471879827s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.601568361 (518h17m0.472651406s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602256040 (518h17m0.473339075s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602745653 (518h17m0.473828688s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.603827122 (518h17m0.474910167s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.604147301 (518h17m0.475230336s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:02.224895133 (518h17m2.095978178s) [upload=454711]: Next returned nil segment
Dec 27 05:46:02.224920952 (518h17m2.096003997s) [upload=454711]: Waiting for error group managing segments...
Dec 27 05:46:02.611896800 (518h17m2.482979835s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.612665283 (518h17m2.483748328s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.618470057 (518h17m2.489553102s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.627484091 (518h17m2.498567126s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.631205627 (518h17m2.502288663s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.632225459 (518h17m2.503308504s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.635912790 (518h17m2.506995836s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637103788 (518h17m2.508186823s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637619290 (518h17m2.508702335s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.638549071 (518h17m2.509632116s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.852825640 (518h17m2.723908685s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.589407397 (518h17m3.460490442s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660687898 (518h17m3.531770943s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660709108 (518h17m3.531792153s) [seg_pos={0 0}, upload=454711]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:46:03.660958513 (518h17m3.532041548s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.660966819 (518h17m3.532049854s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661122996 (518h17m3.532206041s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661130670 (518h17m3.532213715s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661259536 (518h17m3.532342581s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661266960 (518h17m3.532350005s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661283091 (518h17m3.532366126s) [seg_pos={0 0}, upload=454711]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:46:03.661385787 (518h17m3.532468822s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:46:03.661391798 (518h17m3.532474833s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:46:03.949073754 (518h17m3.820156789s) [upload=454711]: Done uploading segments...
Dec 27 05:46:03.949109782 (518h17m3.820192827s) [upload=454711]: Upload finished. err: <nil>
Duration: 5.702138517s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [......X...............................................X.....]
duration: 
3.653592284s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:49:20 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.653620998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.........................X.............X....................]
duration: 
38.247993434s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:49:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 38.248024482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [.XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X....]
duration: 
5.956130531s
trace: 
https://jaeger.dev.storj.io/trace/e6b9cde4f4af8a7
FAIL (2025-12-27 05:49:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA) context deadline exceeded
======LOGS======
Dec 27 05:46:00.348872485 (518h17m0.21995552s) [upload=454711]: Starting upload
Dec 27 05:46:00.348967666 (518h17m0.220050711s) [upload=454711]: Done starting upload
Dec 27 05:46:00.349066805 (518h17m0.22014984s) [upload=454711]: Uploading segments...
Dec 27 05:46:00.349230336 (518h17m0.220313381s) [upload=454711]: Got next segment. Inline: false
Dec 27 05:46:00.349284158 (518h17m0.220367193s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:46:00.349289890 (518h17m0.220372935s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:46:00.596547460 (518h17m0.467630515s) [seg_pos={0 0}, upload=454711]: Begin upload segment...
Dec 27 05:46:00.596600422 (518h17m0.467683467s) [seg_pos={0 0}, upload=454711]: Done begin upload segment.
Dec 27 05:46:00.596626500 (518h17m0.467709535s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596638704 (518h17m0.467721749s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596833695 (518h17m0.46791675s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597223769 (518h17m0.468306814s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597563425 (518h17m0.46864646s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598011227 (518h17m0.469094272s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598422290 (518h17m0.469505325s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598792486 (518h17m0.469875531s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.599359966 (518h17m0.470443011s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600416808 (518h17m0.471499843s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600796782 (518h17m0.471879827s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.601568361 (518h17m0.472651406s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602256040 (518h17m0.473339075s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602745653 (518h17m0.473828688s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.603827122 (518h17m0.474910167s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.604147301 (518h17m0.475230336s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:02.224895133 (518h17m2.095978178s) [upload=454711]: Next returned nil segment
Dec 27 05:46:02.224920952 (518h17m2.096003997s) [upload=454711]: Waiting for error group managing segments...
Dec 27 05:46:02.611896800 (518h17m2.482979835s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.612665283 (518h17m2.483748328s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.618470057 (518h17m2.489553102s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.627484091 (518h17m2.498567126s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.631205627 (518h17m2.502288663s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.632225459 (518h17m2.503308504s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.635912790 (518h17m2.506995836s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637103788 (518h17m2.508186823s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637619290 (518h17m2.508702335s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.638549071 (518h17m2.509632116s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.852825640 (518h17m2.723908685s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.589407397 (518h17m3.460490442s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660687898 (518h17m3.531770943s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660709108 (518h17m3.531792153s) [seg_pos={0 0}, upload=454711]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:46:03.660958513 (518h17m3.532041548s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.660966819 (518h17m3.532049854s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661122996 (518h17m3.532206041s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661130670 (518h17m3.532213715s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661259536 (518h17m3.532342581s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661266960 (518h17m3.532350005s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661283091 (518h17m3.532366126s) [seg_pos={0 0}, upload=454711]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:46:03.661385787 (518h17m3.532468822s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:46:03.661391798 (518h17m3.532474833s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:46:03.949073754 (518h17m3.820156789s) [upload=454711]: Done uploading segments...
Dec 27 05:46:03.949109782 (518h17m3.820192827s) [upload=454711]: Upload finished. err: <nil>
Duration: 5.956197739s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [.......X...............................................X....]
duration: 
3.65606336s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:48:50 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.656085672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..........................X.............X...................]
duration: 
35.599358142s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:48:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 35.599385092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X...]
duration: 
5.922886928s
trace: 
https://jaeger.dev.storj.io/trace/e6b9cde4f4af8a7
FAIL (2025-12-27 05:47:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA) context deadline exceeded
======LOGS======
Dec 27 05:46:00.348872485 (518h17m0.21995552s) [upload=454711]: Starting upload
Dec 27 05:46:00.348967666 (518h17m0.220050711s) [upload=454711]: Done starting upload
Dec 27 05:46:00.349066805 (518h17m0.22014984s) [upload=454711]: Uploading segments...
Dec 27 05:46:00.349230336 (518h17m0.220313381s) [upload=454711]: Got next segment. Inline: false
Dec 27 05:46:00.349284158 (518h17m0.220367193s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:46:00.349289890 (518h17m0.220372935s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:46:00.596547460 (518h17m0.467630515s) [seg_pos={0 0}, upload=454711]: Begin upload segment...
Dec 27 05:46:00.596600422 (518h17m0.467683467s) [seg_pos={0 0}, upload=454711]: Done begin upload segment.
Dec 27 05:46:00.596626500 (518h17m0.467709535s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596638704 (518h17m0.467721749s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596833695 (518h17m0.46791675s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597223769 (518h17m0.468306814s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597563425 (518h17m0.46864646s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598011227 (518h17m0.469094272s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598422290 (518h17m0.469505325s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598792486 (518h17m0.469875531s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.599359966 (518h17m0.470443011s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600416808 (518h17m0.471499843s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600796782 (518h17m0.471879827s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.601568361 (518h17m0.472651406s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602256040 (518h17m0.473339075s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602745653 (518h17m0.473828688s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.603827122 (518h17m0.474910167s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.604147301 (518h17m0.475230336s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:02.224895133 (518h17m2.095978178s) [upload=454711]: Next returned nil segment
Dec 27 05:46:02.224920952 (518h17m2.096003997s) [upload=454711]: Waiting for error group managing segments...
Dec 27 05:46:02.611896800 (518h17m2.482979835s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.612665283 (518h17m2.483748328s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.618470057 (518h17m2.489553102s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.627484091 (518h17m2.498567126s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.631205627 (518h17m2.502288663s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.632225459 (518h17m2.503308504s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.635912790 (518h17m2.506995836s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637103788 (518h17m2.508186823s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637619290 (518h17m2.508702335s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.638549071 (518h17m2.509632116s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.852825640 (518h17m2.723908685s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.589407397 (518h17m3.460490442s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660687898 (518h17m3.531770943s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660709108 (518h17m3.531792153s) [seg_pos={0 0}, upload=454711]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:46:03.660958513 (518h17m3.532041548s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.660966819 (518h17m3.532049854s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661122996 (518h17m3.532206041s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661130670 (518h17m3.532213715s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661259536 (518h17m3.532342581s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661266960 (518h17m3.532350005s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661283091 (518h17m3.532366126s) [seg_pos={0 0}, upload=454711]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:46:03.661385787 (518h17m3.532468822s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:46:03.661391798 (518h17m3.532474833s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:46:03.949073754 (518h17m3.820156789s) [upload=454711]: Done uploading segments...
Dec 27 05:46:03.949109782 (518h17m3.820192827s) [upload=454711]: Upload finished. err: <nil>
Duration: 5.922924319s core uplink us-select-1