Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 896626 reports (Page 11375 of 17933)
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [........X...............................................X...]
duration: 
3.55526895s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:47:52 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.555295178s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [.........X...............................................X..]
duration: 
3.614393474s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:47:27 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.614651784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [............................X.............X.................]
duration: 
47.021419518s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:47:06 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 47.021451118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [X..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X..]
duration: 
5.990563158s
trace: 
https://jaeger.dev.storj.io/trace/e6b9cde4f4af8a7
FAIL (2025-12-27 05:46:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA) context deadline exceeded
======LOGS======
Dec 27 05:46:00.348872485 (518h17m0.21995552s) [upload=454711]: Starting upload
Dec 27 05:46:00.348967666 (518h17m0.220050711s) [upload=454711]: Done starting upload
Dec 27 05:46:00.349066805 (518h17m0.22014984s) [upload=454711]: Uploading segments...
Dec 27 05:46:00.349230336 (518h17m0.220313381s) [upload=454711]: Got next segment. Inline: false
Dec 27 05:46:00.349284158 (518h17m0.220367193s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:46:00.349289890 (518h17m0.220372935s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:46:00.596547460 (518h17m0.467630515s) [seg_pos={0 0}, upload=454711]: Begin upload segment...
Dec 27 05:46:00.596600422 (518h17m0.467683467s) [seg_pos={0 0}, upload=454711]: Done begin upload segment.
Dec 27 05:46:00.596626500 (518h17m0.467709535s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596638704 (518h17m0.467721749s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596833695 (518h17m0.46791675s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597223769 (518h17m0.468306814s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597563425 (518h17m0.46864646s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598011227 (518h17m0.469094272s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598422290 (518h17m0.469505325s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598792486 (518h17m0.469875531s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.599359966 (518h17m0.470443011s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600416808 (518h17m0.471499843s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600796782 (518h17m0.471879827s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.601568361 (518h17m0.472651406s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602256040 (518h17m0.473339075s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602745653 (518h17m0.473828688s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.603827122 (518h17m0.474910167s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.604147301 (518h17m0.475230336s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:02.224895133 (518h17m2.095978178s) [upload=454711]: Next returned nil segment
Dec 27 05:46:02.224920952 (518h17m2.096003997s) [upload=454711]: Waiting for error group managing segments...
Dec 27 05:46:02.611896800 (518h17m2.482979835s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.612665283 (518h17m2.483748328s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.618470057 (518h17m2.489553102s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.627484091 (518h17m2.498567126s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.631205627 (518h17m2.502288663s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.632225459 (518h17m2.503308504s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.635912790 (518h17m2.506995836s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637103788 (518h17m2.508186823s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637619290 (518h17m2.508702335s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.638549071 (518h17m2.509632116s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.852825640 (518h17m2.723908685s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.589407397 (518h17m3.460490442s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660687898 (518h17m3.531770943s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660709108 (518h17m3.531792153s) [seg_pos={0 0}, upload=454711]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:46:03.660958513 (518h17m3.532041548s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.660966819 (518h17m3.532049854s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661122996 (518h17m3.532206041s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661130670 (518h17m3.532213715s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661259536 (518h17m3.532342581s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661266960 (518h17m3.532350005s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661283091 (518h17m3.532366126s) [seg_pos={0 0}, upload=454711]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:46:03.661385787 (518h17m3.532468822s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:46:03.661391798 (518h17m3.532474833s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:46:03.949073754 (518h17m3.820156789s) [upload=454711]: Done uploading segments...
Dec 27 05:46:03.949109782 (518h17m3.820192827s) [upload=454711]: Upload finished. err: <nil>
Duration: 5.99059606s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [..........X...............................................X.]
duration: 
3.527966774s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:46:49 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.527995158s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X.]
duration: 
5.704545048s
trace: 
https://jaeger.dev.storj.io/trace/e6b9cde4f4af8a7
FAIL (2025-12-27 05:46:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA) context deadline exceeded
======LOGS======
Dec 27 05:46:00.348872485 (518h17m0.21995552s) [upload=454711]: Starting upload
Dec 27 05:46:00.348967666 (518h17m0.220050711s) [upload=454711]: Done starting upload
Dec 27 05:46:00.349066805 (518h17m0.22014984s) [upload=454711]: Uploading segments...
Dec 27 05:46:00.349230336 (518h17m0.220313381s) [upload=454711]: Got next segment. Inline: false
Dec 27 05:46:00.349284158 (518h17m0.220367193s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:46:00.349289890 (518h17m0.220372935s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:46:00.596547460 (518h17m0.467630515s) [seg_pos={0 0}, upload=454711]: Begin upload segment...
Dec 27 05:46:00.596600422 (518h17m0.467683467s) [seg_pos={0 0}, upload=454711]: Done begin upload segment.
Dec 27 05:46:00.596626500 (518h17m0.467709535s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596638704 (518h17m0.467721749s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596833695 (518h17m0.46791675s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597223769 (518h17m0.468306814s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597563425 (518h17m0.46864646s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598011227 (518h17m0.469094272s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598422290 (518h17m0.469505325s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598792486 (518h17m0.469875531s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.599359966 (518h17m0.470443011s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600416808 (518h17m0.471499843s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600796782 (518h17m0.471879827s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.601568361 (518h17m0.472651406s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602256040 (518h17m0.473339075s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602745653 (518h17m0.473828688s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.603827122 (518h17m0.474910167s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.604147301 (518h17m0.475230336s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:02.224895133 (518h17m2.095978178s) [upload=454711]: Next returned nil segment
Dec 27 05:46:02.224920952 (518h17m2.096003997s) [upload=454711]: Waiting for error group managing segments...
Dec 27 05:46:02.611896800 (518h17m2.482979835s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.612665283 (518h17m2.483748328s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.618470057 (518h17m2.489553102s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.627484091 (518h17m2.498567126s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.631205627 (518h17m2.502288663s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.632225459 (518h17m2.503308504s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.635912790 (518h17m2.506995836s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637103788 (518h17m2.508186823s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637619290 (518h17m2.508702335s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.638549071 (518h17m2.509632116s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.852825640 (518h17m2.723908685s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.589407397 (518h17m3.460490442s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660687898 (518h17m3.531770943s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660709108 (518h17m3.531792153s) [seg_pos={0 0}, upload=454711]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:46:03.660958513 (518h17m3.532041548s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.660966819 (518h17m3.532049854s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661122996 (518h17m3.532206041s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661130670 (518h17m3.532213715s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661259536 (518h17m3.532342581s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661266960 (518h17m3.532350005s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661283091 (518h17m3.532366126s) [seg_pos={0 0}, upload=454711]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:46:03.661385787 (518h17m3.532468822s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:46:03.661391798 (518h17m3.532474833s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:46:03.949073754 (518h17m3.820156789s) [upload=454711]: Done uploading segments...
Dec 27 05:46:03.949109782 (518h17m3.820192827s) [upload=454711]: Upload finished. err: <nil>
Duration: 5.704695444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.X]
duration: 
30.003661062s
trace: 
https://jaeger.dev.storj.io/trace/e6b9cde4f4af8a7
FAIL (2025-12-27 05:46:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA) context deadline exceeded
======LOGS======
Dec 27 05:46:00.348872485 (518h17m0.21995552s) [upload=454711]: Starting upload
Dec 27 05:46:00.348967666 (518h17m0.220050711s) [upload=454711]: Done starting upload
Dec 27 05:46:00.349066805 (518h17m0.22014984s) [upload=454711]: Uploading segments...
Dec 27 05:46:00.349230336 (518h17m0.220313381s) [upload=454711]: Got next segment. Inline: false
Dec 27 05:46:00.349284158 (518h17m0.220367193s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:46:00.349289890 (518h17m0.220372935s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:46:00.596547460 (518h17m0.467630515s) [seg_pos={0 0}, upload=454711]: Begin upload segment...
Dec 27 05:46:00.596600422 (518h17m0.467683467s) [seg_pos={0 0}, upload=454711]: Done begin upload segment.
Dec 27 05:46:00.596626500 (518h17m0.467709535s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596638704 (518h17m0.467721749s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.596833695 (518h17m0.46791675s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597223769 (518h17m0.468306814s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.597563425 (518h17m0.46864646s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598011227 (518h17m0.469094272s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598422290 (518h17m0.469505325s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.598792486 (518h17m0.469875531s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.599359966 (518h17m0.470443011s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600416808 (518h17m0.471499843s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.600796782 (518h17m0.471879827s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.601568361 (518h17m0.472651406s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602256040 (518h17m0.473339075s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.602745653 (518h17m0.473828688s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.603827122 (518h17m0.474910167s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:00.604147301 (518h17m0.475230336s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Uploading piece...
Dec 27 05:46:02.224895133 (518h17m2.095978178s) [upload=454711]: Next returned nil segment
Dec 27 05:46:02.224920952 (518h17m2.096003997s) [upload=454711]: Waiting for error group managing segments...
Dec 27 05:46:02.611896800 (518h17m2.482979835s) [noise=true , address=145.79.227.42:4121   , piece_id=PJNRR3OYQOTY5P7CBOTTDZLPZTWU6WVO6NMRIFOMDBRAXPUISKNQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.612665283 (518h17m2.483748328s) [noise=true , address=145.79.227.49:4339   , piece_id=OLRAA77JQJZ7EZA4UAWSMIV72UKPDIIDB2LINOONZJZ2YQD53QSA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.618470057 (518h17m2.489553102s) [noise=true , address=145.79.227.46:4601   , piece_id=UPCGJGGNLSTDGPRBH5E6WZB3I6RC7UHYB6IDLPE5KUSLAUAWWDPA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.627484091 (518h17m2.498567126s) [noise=true , address=145.79.227.11:4732   , piece_id=CR75UCIDD74HEHQVI46AGWCWNRXOFFNF5SVOQAT247NMSJITVDFQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.631205627 (518h17m2.502288663s) [noise=true , address=145.79.227.9:4366    , piece_id=7N7ST6U3DWOLKICZD6VSJYFUI5OQZIYWY6EQNXEKPRVDY53JZHVA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.632225459 (518h17m2.503308504s) [noise=true , address=145.79.227.31:4798   , piece_id=LKYLIK7BN6A2LR5G34CTSHJQUYDZU4BSZSZGVZTDHDQATLMXO65Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.635912790 (518h17m2.506995836s) [noise=true , address=145.79.227.38:4979   , piece_id=WB5TBZPFE2M6HKKOYZLYEIEPN66DEMDWV6BPUPDFC6G7JAZ2AMDQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637103788 (518h17m2.508186823s) [noise=true , address=145.79.227.4:4185    , piece_id=SOR2IRGQ2CXUUVJRHGAUA575SQVLDMTXQFGILIQLVZS3TM7J27GQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.637619290 (518h17m2.508702335s) [noise=true , address=145.79.227.47:4265   , piece_id=YNBNOTQTKJJB2FI6MAFXCBHKVVKF7WV3UDGLPTRPVRSCXU6DSCVQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.638549071 (518h17m2.509632116s) [noise=true , address=145.79.227.17:4971   , piece_id=OVIT7PEI5MZVFVKXQHBSSSDXIPPGM4BZ26N52U2ZFAS2DSGXPQ3Q, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:02.852825640 (518h17m2.723908685s) [noise=true , address=45.91.116.4:4940     , piece_id=SRZB52TWDKYKKEZSCJ6H74W7LPW2NYXM4XW5IFAXAOIZHC6N7XJQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.589407397 (518h17m3.460490442s) [noise=true , address=45.91.116.11:4118    , piece_id=TJ7AP4ZWHLQQKB4VCOTZVNNT35XBXUPKI7RL7F7NXKGFYFQ7FHUA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660687898 (518h17m3.531770943s) [noise=true , address=45.91.116.9:4261     , piece_id=OOSJIERDFD3XKRN3A4W7YIH3ZFURPJQPHBWBPPUPUFZYORTMVYBQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: <nil>
Dec 27 05:46:03.660709108 (518h17m3.531792153s) [seg_pos={0 0}, upload=454711]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:46:03.660958513 (518h17m3.532041548s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.660966819 (518h17m3.532049854s) [noise=true , address=77.90.31.10:4518     , piece_id=W5DZ7ADSTGWTVI772Y7NGV4ESXALSOX5TP6NXVRCHPOPQ76KGYAA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661122996 (518h17m3.532206041s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661130670 (518h17m3.532213715s) [noise=true , address=77.90.31.16:4735     , piece_id=2WP6NR5UVT2F2LDFLQ7ZDMQGQJDGTL6YVTIF6LPDNM55UOSOFTEA, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661259536 (518h17m3.532342581s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 27 05:46:03.661266960 (518h17m3.532350005s) [noise=true , address=45.91.116.14:4191    , piece_id=RHXWHTQAHHODSUUPLCSRBRGZY5XS7BWRJTGWQEFQQRST36BSWDIQ, seg_pos={0 0}, upload=454711]: Long tail cancellation due to optimal threshold
Dec 27 05:46:03.661283091 (518h17m3.532366126s) [seg_pos={0 0}, upload=454711]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:46:03.661385787 (518h17m3.532468822s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:46:03.661391798 (518h17m3.532474833s) [upload=454711]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:46:03.949073754 (518h17m3.820156789s) [upload=454711]: Done uploading segments...
Dec 27 05:46:03.949109782 (518h17m3.820192827s) [upload=454711]: Upload finished. err: <nil>
Duration: 30.003683025s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............................X.............X................]
duration: 
44.066532991s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:46:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 44.066559221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [...........X...............................................X]
duration: 
5.001111471s
trace: 
https://jaeger.dev.storj.io/trace/39d952b81959c79e
FAIL (2025-12-27 05:45:49 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001123874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X.]
duration: 
26.931986354s
trace: 
https://jaeger.dev.storj.io/trace/c8006671d64a072
FAIL (2025-12-27 05:45:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: YL7IX4RTUHYE3PPG72EQUCNKOVFA7ISJQO33OCNZ57LB2YHOIEKA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: FRQC3RSQHLYBXU7DR6BOGU7GCQWR4QFQICP37AK5RWMWPUPTIE7A) context deadline exceeded
======LOGS======
Dec 27 05:44:45.346609266 (518h15m45.217692311s) [upload=454692]: Starting upload
Dec 27 05:44:45.346687966 (518h15m45.217771011s) [upload=454692]: Done starting upload
Dec 27 05:44:45.346859052 (518h15m45.217942107s) [upload=454692]: Uploading segments...
Dec 27 05:44:45.347073129 (518h15m45.218156244s) [upload=454692]: Got next segment. Inline: false
Dec 27 05:44:45.347107244 (518h15m45.218190279s) [upload=454692]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:44:45.347112816 (518h15m45.218195861s) [upload=454692]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:44:45.592588152 (518h15m45.463671187s) [seg_pos={0 0}, upload=454692]: Begin upload segment...
Dec 27 05:44:45.592631745 (518h15m45.46371478s) [seg_pos={0 0}, upload=454692]: Done begin upload segment.
Dec 27 05:44:45.592655100 (518h15m45.463738135s) [noise=true , address=45.91.116.6:4976     , piece_id=CHY6URMZZ33A3JJYMJQL7OGO7PY6SOMWMNLYJI6RC22S3JMBJAKQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.596423676 (518h15m45.467506711s) [noise=true , address=145.79.227.38:4977   , piece_id=JEKWFTETVI3NVZNMLFQMKHJCHKYZSQPGH67RPWHFWODOSKF6T6TA, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.597688926 (518h15m45.468771961s) [noise=true , address=145.79.227.15:4873   , piece_id=55A5WBQX7NEEHVL2CQUVGYPZZQBCVCKHQ2K64ZKWL5SWQO35EHUQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.598976167 (518h15m45.470059202s) [noise=true , address=77.90.31.10:4520     , piece_id=YL7IX4RTUHYE3PPG72EQUCNKOVFA7ISJQO33OCNZ57LB2YHOIEKA, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.600272144 (518h15m45.471355189s) [noise=true , address=145.79.227.6:4257    , piece_id=UKKYII5DYSFMSGDJSX6AGLDHDPT4Q7TQ5KFLL5QJXRDKEPYIJOIA, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.601504330 (518h15m45.472587405s) [noise=true , address=145.79.227.46:4602   , piece_id=SLXEEPFVVKKKXYZZ5IXNTVNUPYDWRLW4SQG6UJ2LFSRH7JCFWUDA, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.602931337 (518h15m45.474014383s) [noise=true , address=145.79.227.34:4907   , piece_id=W7IUOOFEDKQBSDZTLRBC2RYRG4PN63HT6DGHA34XHNNSQJW3WI5A, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.604133126 (518h15m45.475216161s) [noise=true , address=145.79.227.3:4151    , piece_id=4M2B27IEA7P6VSS7GMAX5NYSCFGNBUXQI6EBWGWPZPJ6X7O7CO4A, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.605681735 (518h15m45.47676478s) [noise=true , address=45.91.116.8:4048     , piece_id=FRQC3RSQHLYBXU7DR6BOGU7GCQWR4QFQICP37AK5RWMWPUPTIE7A, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.609514914 (518h15m45.480597949s) [noise=true , address=77.90.31.21:4877     , piece_id=ORAWKVQGIHWND3H7EJ6ROLPDDDGF4HVO3O7CSE5FRBOHZ7J2VLMQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.615154674 (518h15m45.486237719s) [noise=true , address=145.79.227.49:4338   , piece_id=GFXSBVXVA5BUDOYNV6VK5HUFJLJHVGXNKCQYA5TW2CYGZYHLSC2Q, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.617949135 (518h15m45.48903217s) [noise=true , address=145.79.227.35:4482   , piece_id=USIBTQUQ4XMJ4DPH4DKG5RYOHX6HP67MZ4CF6W7GSPL3FXX46ZNQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.624289988 (518h15m45.495373024s) [noise=true , address=45.91.116.4:4939     , piece_id=NLDXHHOC3K5H6EJDG5M457JXAZQSPPT7CPQUDS4TCXCS7GGLMXLQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.625692729 (518h15m45.496775764s) [noise=true , address=77.90.31.14:4661     , piece_id=U36ZTMSLSB7G5HHEHCEG7AI4733HSGGNHNVCZBXDSO6Y5XFGPE2Q, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.626797934 (518h15m45.497880969s) [noise=true , address=145.79.227.19:4077   , piece_id=DVW3VIV5CJTB26Z4QT6SPQRD7Z7UWPNZOG4TUOU3ABMALPNLKQ3A, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.627973092 (518h15m45.499056127s) [noise=true , address=145.79.227.16:4909   , piece_id=GNHW7Y5XGPTR7WHUTFNWOHZHHGZNCYGGHLT6S4ZHGKTB3ITYK3VQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:47.359537456 (518h15m47.230620491s) [upload=454692]: Next returned nil segment
Dec 27 05:44:47.359552344 (518h15m47.230635379s) [upload=454692]: Waiting for error group managing segments...
Dec 27 05:44:47.643430828 (518h15m47.514513873s) [noise=true , address=145.79.227.15:4873   , piece_id=55A5WBQX7NEEHVL2CQUVGYPZZQBCVCKHQ2K64ZKWL5SWQO35EHUQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.644366108 (518h15m47.515449153s) [noise=true , address=145.79.227.49:4338   , piece_id=GFXSBVXVA5BUDOYNV6VK5HUFJLJHVGXNKCQYA5TW2CYGZYHLSC2Q, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.648570233 (518h15m47.519653279s) [noise=true , address=145.79.227.3:4151    , piece_id=4M2B27IEA7P6VSS7GMAX5NYSCFGNBUXQI6EBWGWPZPJ6X7O7CO4A, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.649780198 (518h15m47.520863233s) [noise=true , address=145.79.227.46:4602   , piece_id=SLXEEPFVVKKKXYZZ5IXNTVNUPYDWRLW4SQG6UJ2LFSRH7JCFWUDA, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.662950426 (518h15m47.534033461s) [noise=true , address=145.79.227.6:4257    , piece_id=UKKYII5DYSFMSGDJSX6AGLDHDPT4Q7TQ5KFLL5QJXRDKEPYIJOIA, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.666647956 (518h15m47.537731001s) [noise=true , address=145.79.227.38:4977   , piece_id=JEKWFTETVI3NVZNMLFQMKHJCHKYZSQPGH67RPWHFWODOSKF6T6TA, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.685912119 (518h15m47.556995154s) [noise=true , address=145.79.227.35:4482   , piece_id=USIBTQUQ4XMJ4DPH4DKG5RYOHX6HP67MZ4CF6W7GSPL3FXX46ZNQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.686653410 (518h15m47.557736466s) [noise=true , address=145.79.227.19:4077   , piece_id=DVW3VIV5CJTB26Z4QT6SPQRD7Z7UWPNZOG4TUOU3ABMALPNLKQ3A, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.703208845 (518h15m47.57429188s) [noise=true , address=145.79.227.16:4909   , piece_id=GNHW7Y5XGPTR7WHUTFNWOHZHHGZNCYGGHLT6S4ZHGKTB3ITYK3VQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.857127212 (518h15m47.728210247s) [noise=true , address=77.90.31.14:4661     , piece_id=U36ZTMSLSB7G5HHEHCEG7AI4733HSGGNHNVCZBXDSO6Y5XFGPE2Q, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.920980750 (518h15m47.792063785s) [noise=true , address=45.91.116.4:4939     , piece_id=NLDXHHOC3K5H6EJDG5M457JXAZQSPPT7CPQUDS4TCXCS7GGLMXLQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:48.263041202 (518h15m48.134124237s) [noise=true , address=45.91.116.8:4048     , piece_id=FRQC3RSQHLYBXU7DR6BOGU7GCQWR4QFQICP37AK5RWMWPUPTIE7A, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:48.487029834 (518h15m48.358112869s) [noise=true , address=77.90.31.10:4520     , piece_id=YL7IX4RTUHYE3PPG72EQUCNKOVFA7ISJQO33OCNZ57LB2YHOIEKA, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:48.487047487 (518h15m48.358130532s) [seg_pos={0 0}, upload=454692]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:44:48.487237009 (518h15m48.358320044s) [noise=true , address=77.90.31.21:4877     , piece_id=ORAWKVQGIHWND3H7EJ6ROLPDDDGF4HVO3O7CSE5FRBOHZ7J2VLMQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 27 05:44:48.487244192 (518h15m48.358327237s) [noise=true , address=77.90.31.21:4877     , piece_id=ORAWKVQGIHWND3H7EJ6ROLPDDDGF4HVO3O7CSE5FRBOHZ7J2VLMQ, seg_pos={0 0}, upload=454692]: Long tail cancellation due to optimal threshold
Dec 27 05:44:48.487457638 (518h15m48.358540673s) [noise=true , address=145.79.227.34:4907   , piece_id=W7IUOOFEDKQBSDZTLRBC2RYRG4PN63HT6DGHA34XHNNSQJW3WI5A, seg_pos={0 0}, upload=454692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 27 05:44:48.487463590 (518h15m48.358546635s) [noise=true , address=145.79.227.34:4907   , piece_id=W7IUOOFEDKQBSDZTLRBC2RYRG4PN63HT6DGHA34XHNNSQJW3WI5A, seg_pos={0 0}, upload=454692]: Long tail cancellation due to optimal threshold
Dec 27 05:44:48.487577838 (518h15m48.358660873s) [noise=true , address=45.91.116.6:4976     , piece_id=CHY6URMZZ33A3JJYMJQL7OGO7PY6SOMWMNLYJI6RC22S3JMBJAKQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 05:44:48.487582857 (518h15m48.358665892s) [noise=true , address=45.91.116.6:4976     , piece_id=CHY6URMZZ33A3JJYMJQL7OGO7PY6SOMWMNLYJI6RC22S3JMBJAKQ, seg_pos={0 0}, upload=454692]: Long tail cancellation due to optimal threshold
Dec 27 05:44:48.487598667 (518h15m48.358681702s) [seg_pos={0 0}, upload=454692]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:44:48.487659683 (518h15m48.358742718s) [upload=454692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:44:48.487664382 (518h15m48.358747417s) [upload=454692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:44:48.777295600 (518h15m48.648378665s) [upload=454692]: Done uploading segments...
Dec 27 05:44:48.777323483 (518h15m48.648406518s) [upload=454692]: Upload finished. err: <nil>
Duration: 26.932020621s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.X]
duration: 
30.004360943s
trace: 
https://jaeger.dev.storj.io/trace/c8006671d64a072
FAIL (2025-12-27 05:45:14 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: YL7IX4RTUHYE3PPG72EQUCNKOVFA7ISJQO33OCNZ57LB2YHOIEKA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: FRQC3RSQHLYBXU7DR6BOGU7GCQWR4QFQICP37AK5RWMWPUPTIE7A) context deadline exceeded
======LOGS======
Dec 27 05:44:45.346609266 (518h15m45.217692311s) [upload=454692]: Starting upload
Dec 27 05:44:45.346687966 (518h15m45.217771011s) [upload=454692]: Done starting upload
Dec 27 05:44:45.346859052 (518h15m45.217942107s) [upload=454692]: Uploading segments...
Dec 27 05:44:45.347073129 (518h15m45.218156244s) [upload=454692]: Got next segment. Inline: false
Dec 27 05:44:45.347107244 (518h15m45.218190279s) [upload=454692]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:44:45.347112816 (518h15m45.218195861s) [upload=454692]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:44:45.592588152 (518h15m45.463671187s) [seg_pos={0 0}, upload=454692]: Begin upload segment...
Dec 27 05:44:45.592631745 (518h15m45.46371478s) [seg_pos={0 0}, upload=454692]: Done begin upload segment.
Dec 27 05:44:45.592655100 (518h15m45.463738135s) [noise=true , address=45.91.116.6:4976     , piece_id=CHY6URMZZ33A3JJYMJQL7OGO7PY6SOMWMNLYJI6RC22S3JMBJAKQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.596423676 (518h15m45.467506711s) [noise=true , address=145.79.227.38:4977   , piece_id=JEKWFTETVI3NVZNMLFQMKHJCHKYZSQPGH67RPWHFWODOSKF6T6TA, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.597688926 (518h15m45.468771961s) [noise=true , address=145.79.227.15:4873   , piece_id=55A5WBQX7NEEHVL2CQUVGYPZZQBCVCKHQ2K64ZKWL5SWQO35EHUQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.598976167 (518h15m45.470059202s) [noise=true , address=77.90.31.10:4520     , piece_id=YL7IX4RTUHYE3PPG72EQUCNKOVFA7ISJQO33OCNZ57LB2YHOIEKA, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.600272144 (518h15m45.471355189s) [noise=true , address=145.79.227.6:4257    , piece_id=UKKYII5DYSFMSGDJSX6AGLDHDPT4Q7TQ5KFLL5QJXRDKEPYIJOIA, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.601504330 (518h15m45.472587405s) [noise=true , address=145.79.227.46:4602   , piece_id=SLXEEPFVVKKKXYZZ5IXNTVNUPYDWRLW4SQG6UJ2LFSRH7JCFWUDA, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.602931337 (518h15m45.474014383s) [noise=true , address=145.79.227.34:4907   , piece_id=W7IUOOFEDKQBSDZTLRBC2RYRG4PN63HT6DGHA34XHNNSQJW3WI5A, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.604133126 (518h15m45.475216161s) [noise=true , address=145.79.227.3:4151    , piece_id=4M2B27IEA7P6VSS7GMAX5NYSCFGNBUXQI6EBWGWPZPJ6X7O7CO4A, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.605681735 (518h15m45.47676478s) [noise=true , address=45.91.116.8:4048     , piece_id=FRQC3RSQHLYBXU7DR6BOGU7GCQWR4QFQICP37AK5RWMWPUPTIE7A, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.609514914 (518h15m45.480597949s) [noise=true , address=77.90.31.21:4877     , piece_id=ORAWKVQGIHWND3H7EJ6ROLPDDDGF4HVO3O7CSE5FRBOHZ7J2VLMQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.615154674 (518h15m45.486237719s) [noise=true , address=145.79.227.49:4338   , piece_id=GFXSBVXVA5BUDOYNV6VK5HUFJLJHVGXNKCQYA5TW2CYGZYHLSC2Q, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.617949135 (518h15m45.48903217s) [noise=true , address=145.79.227.35:4482   , piece_id=USIBTQUQ4XMJ4DPH4DKG5RYOHX6HP67MZ4CF6W7GSPL3FXX46ZNQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.624289988 (518h15m45.495373024s) [noise=true , address=45.91.116.4:4939     , piece_id=NLDXHHOC3K5H6EJDG5M457JXAZQSPPT7CPQUDS4TCXCS7GGLMXLQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.625692729 (518h15m45.496775764s) [noise=true , address=77.90.31.14:4661     , piece_id=U36ZTMSLSB7G5HHEHCEG7AI4733HSGGNHNVCZBXDSO6Y5XFGPE2Q, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.626797934 (518h15m45.497880969s) [noise=true , address=145.79.227.19:4077   , piece_id=DVW3VIV5CJTB26Z4QT6SPQRD7Z7UWPNZOG4TUOU3ABMALPNLKQ3A, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:45.627973092 (518h15m45.499056127s) [noise=true , address=145.79.227.16:4909   , piece_id=GNHW7Y5XGPTR7WHUTFNWOHZHHGZNCYGGHLT6S4ZHGKTB3ITYK3VQ, seg_pos={0 0}, upload=454692]: Uploading piece...
Dec 27 05:44:47.359537456 (518h15m47.230620491s) [upload=454692]: Next returned nil segment
Dec 27 05:44:47.359552344 (518h15m47.230635379s) [upload=454692]: Waiting for error group managing segments...
Dec 27 05:44:47.643430828 (518h15m47.514513873s) [noise=true , address=145.79.227.15:4873   , piece_id=55A5WBQX7NEEHVL2CQUVGYPZZQBCVCKHQ2K64ZKWL5SWQO35EHUQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.644366108 (518h15m47.515449153s) [noise=true , address=145.79.227.49:4338   , piece_id=GFXSBVXVA5BUDOYNV6VK5HUFJLJHVGXNKCQYA5TW2CYGZYHLSC2Q, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.648570233 (518h15m47.519653279s) [noise=true , address=145.79.227.3:4151    , piece_id=4M2B27IEA7P6VSS7GMAX5NYSCFGNBUXQI6EBWGWPZPJ6X7O7CO4A, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.649780198 (518h15m47.520863233s) [noise=true , address=145.79.227.46:4602   , piece_id=SLXEEPFVVKKKXYZZ5IXNTVNUPYDWRLW4SQG6UJ2LFSRH7JCFWUDA, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.662950426 (518h15m47.534033461s) [noise=true , address=145.79.227.6:4257    , piece_id=UKKYII5DYSFMSGDJSX6AGLDHDPT4Q7TQ5KFLL5QJXRDKEPYIJOIA, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.666647956 (518h15m47.537731001s) [noise=true , address=145.79.227.38:4977   , piece_id=JEKWFTETVI3NVZNMLFQMKHJCHKYZSQPGH67RPWHFWODOSKF6T6TA, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.685912119 (518h15m47.556995154s) [noise=true , address=145.79.227.35:4482   , piece_id=USIBTQUQ4XMJ4DPH4DKG5RYOHX6HP67MZ4CF6W7GSPL3FXX46ZNQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.686653410 (518h15m47.557736466s) [noise=true , address=145.79.227.19:4077   , piece_id=DVW3VIV5CJTB26Z4QT6SPQRD7Z7UWPNZOG4TUOU3ABMALPNLKQ3A, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.703208845 (518h15m47.57429188s) [noise=true , address=145.79.227.16:4909   , piece_id=GNHW7Y5XGPTR7WHUTFNWOHZHHGZNCYGGHLT6S4ZHGKTB3ITYK3VQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.857127212 (518h15m47.728210247s) [noise=true , address=77.90.31.14:4661     , piece_id=U36ZTMSLSB7G5HHEHCEG7AI4733HSGGNHNVCZBXDSO6Y5XFGPE2Q, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:47.920980750 (518h15m47.792063785s) [noise=true , address=45.91.116.4:4939     , piece_id=NLDXHHOC3K5H6EJDG5M457JXAZQSPPT7CPQUDS4TCXCS7GGLMXLQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:48.263041202 (518h15m48.134124237s) [noise=true , address=45.91.116.8:4048     , piece_id=FRQC3RSQHLYBXU7DR6BOGU7GCQWR4QFQICP37AK5RWMWPUPTIE7A, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:48.487029834 (518h15m48.358112869s) [noise=true , address=77.90.31.10:4520     , piece_id=YL7IX4RTUHYE3PPG72EQUCNKOVFA7ISJQO33OCNZ57LB2YHOIEKA, seg_pos={0 0}, upload=454692]: Done uploading piece. err: <nil>
Dec 27 05:44:48.487047487 (518h15m48.358130532s) [seg_pos={0 0}, upload=454692]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:44:48.487237009 (518h15m48.358320044s) [noise=true , address=77.90.31.21:4877     , piece_id=ORAWKVQGIHWND3H7EJ6ROLPDDDGF4HVO3O7CSE5FRBOHZ7J2VLMQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF
Dec 27 05:44:48.487244192 (518h15m48.358327237s) [noise=true , address=77.90.31.21:4877     , piece_id=ORAWKVQGIHWND3H7EJ6ROLPDDDGF4HVO3O7CSE5FRBOHZ7J2VLMQ, seg_pos={0 0}, upload=454692]: Long tail cancellation due to optimal threshold
Dec 27 05:44:48.487457638 (518h15m48.358540673s) [noise=true , address=145.79.227.34:4907   , piece_id=W7IUOOFEDKQBSDZTLRBC2RYRG4PN63HT6DGHA34XHNNSQJW3WI5A, seg_pos={0 0}, upload=454692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 27 05:44:48.487463590 (518h15m48.358546635s) [noise=true , address=145.79.227.34:4907   , piece_id=W7IUOOFEDKQBSDZTLRBC2RYRG4PN63HT6DGHA34XHNNSQJW3WI5A, seg_pos={0 0}, upload=454692]: Long tail cancellation due to optimal threshold
Dec 27 05:44:48.487577838 (518h15m48.358660873s) [noise=true , address=45.91.116.6:4976     , piece_id=CHY6URMZZ33A3JJYMJQL7OGO7PY6SOMWMNLYJI6RC22S3JMBJAKQ, seg_pos={0 0}, upload=454692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 05:44:48.487582857 (518h15m48.358665892s) [noise=true , address=45.91.116.6:4976     , piece_id=CHY6URMZZ33A3JJYMJQL7OGO7PY6SOMWMNLYJI6RC22S3JMBJAKQ, seg_pos={0 0}, upload=454692]: Long tail cancellation due to optimal threshold
Dec 27 05:44:48.487598667 (518h15m48.358681702s) [seg_pos={0 0}, upload=454692]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:44:48.487659683 (518h15m48.358742718s) [upload=454692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:44:48.487664382 (518h15m48.358747417s) [upload=454692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:44:48.777295600 (518h15m48.648378665s) [upload=454692]: Done uploading segments...
Dec 27 05:44:48.777323483 (518h15m48.648406518s) [upload=454692]: Upload finished. err: <nil>
Duration: 30.004388325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..............................X.............X...............]
duration: 
45.478813121s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:44:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 45.478855262s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [............X...............................................]
duration: 
3.604723142s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:44:45 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.604749403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX.]
duration: 
5.748026874s
trace: 
https://jaeger.dev.storj.io/trace/854e50d2395b150
FAIL (2025-12-27 05:44:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 3ORUMBAKKRFYX7LDW3JCFNI5X7PBAWDHLDEVBFXJRAGWLNSDQQRQ) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 2GBYGNMRTA7FGHWAGLAPUMIDR7V2JLWLCJ5YMCMH76QDUFQDRDDA) context deadline exceeded
======LOGS======
alse
Dec 27 05:43:46.233969101 (518h14m46.105052136s) [upload=454679]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:43:46.233974202 (518h14m46.105057237s) [upload=454679]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:43:46.495335648 (518h14m46.366418693s) [seg_pos={0 0}, upload=454679]: Begin upload segment...
Dec 27 05:43:46.495397997 (518h14m46.366481042s) [seg_pos={0 0}, upload=454679]: Done begin upload segment.
Dec 27 05:43:46.495430810 (518h14m46.366513845s) [noise=true , address=145.79.227.38:4979   , piece_id=UUIYYFQ2RHXTV6N6MIRHEGKYNKIYR5G74ZOCJ4HMJGWJG767NMGA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.495518446 (518h14m46.366601481s) [noise=true , address=145.79.227.11:4730   , piece_id=3LPHDP4YQQXCPNTR3HFS3MABDRLC2RIQJTINCQ4LNL3WFR5KQZJQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.496208049 (518h14m46.367291094s) [noise=true , address=145.79.227.51:4410   , piece_id=WOU5YUA75E3NDJMPAPT2N3N322EOHC35ZG2IJHETON7RYHKRZVHQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.496555431 (518h14m46.367638466s) [noise=true , address=145.79.227.46:4602   , piece_id=53V5GAJUSFLQTVDTW7Y4TSL6XS52KNGA5RV32HD5CYPYR2KV2TIQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.496872114 (518h14m46.367955149s) [noise=true , address=45.91.116.7:4009     , piece_id=YFMAMBIXNK6EHCQRSYRUZXJFOWHMOQ7E5TXN2EL23PBKGJM7E26Q, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.497191442 (518h14m46.368274477s) [noise=true , address=145.79.227.42:4123   , piece_id=ITUYKRK53RLGUOWQVMZHAHQCQPKU5V4HBANY7UZNC5SLTGWGAJVA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.497523595 (518h14m46.36860664s) [noise=true , address=45.91.116.9:4261     , piece_id=YPJ2YX476XJBGTEXMZCY4KRO4P2URNWISNPJDJQW6MNITC2TAU3A, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.497888901 (518h14m46.368971936s) [noise=true , address=145.79.227.41:4085   , piece_id=JVMHSEAQLGSIVOTYGGAKTJS4M6NDMAT5U73SPYLPG2YKGBK542PA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.499798165 (518h14m46.3708812s) [noise=true , address=45.91.116.12:4228    , piece_id=2GBYGNMRTA7FGHWAGLAPUMIDR7V2JLWLCJ5YMCMH76QDUFQDRDDA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.501363787 (518h14m46.372446822s) [noise=true , address=77.90.31.17:4773     , piece_id=AMERHRT6KPNHF7UT2XZ27UVG6QVAWLZLCXJ5MJHVSZBKDSIQNH7A, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.503219721 (518h14m46.374302766s) [noise=true , address=45.91.116.10:4084    , piece_id=3ORUMBAKKRFYX7LDW3JCFNI5X7PBAWDHLDEVBFXJRAGWLNSDQQRQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.505097717 (518h14m46.376180752s) [noise=true , address=145.79.227.47:4267   , piece_id=DEHV7R3ZXFGOAEMN5D7YI66523XP4SFNQOWV4HWBSEYNXQAE7ILQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.506464860 (518h14m46.377547895s) [noise=true , address=45.91.116.13:4155    , piece_id=OLMLVKBSCVLKUMDCXNJ6YHC73D5ZE3WP2JIBPKSFAIIWFQHSGOYA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.507770506 (518h14m46.378853541s) [noise=true , address=45.91.116.5:4299     , piece_id=DRYGB3BWR3LYA7KLVQSM5P3CEZF5G3QIKYCTSR23LKC5GFKMPSRQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.509028432 (518h14m46.380111467s) [noise=true , address=145.79.227.12:4765   , piece_id=LQMLIEE7FU65KHQ4BNUV5TRRLWM53ZLHPH6JLJJHZ7VVY4HGGVMQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.511188966 (518h14m46.382272001s) [noise=true , address=145.79.227.8:4330    , piece_id=L4A6OWPMYFRN4AWNAV7OJJB2CROYUMVHCWWYOTICN45SV4NDTTFA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:48.202636992 (518h14m48.073720027s) [upload=454679]: Next returned nil segment
Dec 27 05:43:48.202656579 (518h14m48.073739624s) [upload=454679]: Waiting for error group managing segments...
Dec 27 05:43:48.486541836 (518h14m48.357624871s) [noise=true , address=145.79.227.51:4410   , piece_id=WOU5YUA75E3NDJMPAPT2N3N322EOHC35ZG2IJHETON7RYHKRZVHQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.497739949 (518h14m48.368822984s) [noise=true , address=145.79.227.11:4730   , piece_id=3LPHDP4YQQXCPNTR3HFS3MABDRLC2RIQJTINCQ4LNL3WFR5KQZJQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.507876039 (518h14m48.378959114s) [noise=true , address=145.79.227.12:4765   , piece_id=LQMLIEE7FU65KHQ4BNUV5TRRLWM53ZLHPH6JLJJHZ7VVY4HGGVMQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.510024029 (518h14m48.381107124s) [noise=true , address=145.79.227.42:4123   , piece_id=ITUYKRK53RLGUOWQVMZHAHQCQPKU5V4HBANY7UZNC5SLTGWGAJVA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.516459283 (518h14m48.387542338s) [noise=true , address=145.79.227.8:4330    , piece_id=L4A6OWPMYFRN4AWNAV7OJJB2CROYUMVHCWWYOTICN45SV4NDTTFA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.524420804 (518h14m48.395503839s) [noise=true , address=145.79.227.41:4085   , piece_id=JVMHSEAQLGSIVOTYGGAKTJS4M6NDMAT5U73SPYLPG2YKGBK542PA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.524543968 (518h14m48.395627003s) [noise=true , address=145.79.227.46:4602   , piece_id=53V5GAJUSFLQTVDTW7Y4TSL6XS52KNGA5RV32HD5CYPYR2KV2TIQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.524960290 (518h14m48.396043335s) [noise=true , address=145.79.227.38:4979   , piece_id=UUIYYFQ2RHXTV6N6MIRHEGKYNKIYR5G74ZOCJ4HMJGWJG767NMGA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.550140599 (518h14m48.421223634s) [noise=true , address=145.79.227.47:4267   , piece_id=DEHV7R3ZXFGOAEMN5D7YI66523XP4SFNQOWV4HWBSEYNXQAE7ILQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.734873772 (518h14m48.605956807s) [noise=true , address=45.91.116.7:4009     , piece_id=YFMAMBIXNK6EHCQRSYRUZXJFOWHMOQ7E5TXN2EL23PBKGJM7E26Q, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.800739161 (518h14m48.671822196s) [noise=true , address=45.91.116.10:4084    , piece_id=3ORUMBAKKRFYX7LDW3JCFNI5X7PBAWDHLDEVBFXJRAGWLNSDQQRQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.804571548 (518h14m48.675654593s) [noise=true , address=77.90.31.17:4773     , piece_id=AMERHRT6KPNHF7UT2XZ27UVG6QVAWLZLCXJ5MJHVSZBKDSIQNH7A, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.852468723 (518h14m48.723551768s) [noise=true , address=45.91.116.12:4228    , piece_id=2GBYGNMRTA7FGHWAGLAPUMIDR7V2JLWLCJ5YMCMH76QDUFQDRDDA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.852489172 (518h14m48.723572207s) [seg_pos={0 0}, upload=454679]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:43:48.852657062 (518h14m48.723740097s) [noise=true , address=45.91.116.5:4299     , piece_id=DRYGB3BWR3LYA7KLVQSM5P3CEZF5G3QIKYCTSR23LKC5GFKMPSRQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:54952->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:54952->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54952->45.91.116.5:4299: use of closed network connection
Dec 27 05:43:48.852662632 (518h14m48.723745667s) [noise=true , address=45.91.116.5:4299     , piece_id=DRYGB3BWR3LYA7KLVQSM5P3CEZF5G3QIKYCTSR23LKC5GFKMPSRQ, seg_pos={0 0}, upload=454679]: Long tail cancellation due to optimal threshold
Dec 27 05:43:48.852727356 (518h14m48.723810411s) [noise=true , address=45.91.116.9:4261     , piece_id=YPJ2YX476XJBGTEXMZCY4KRO4P2URNWISNPJDJQW6MNITC2TAU3A, seg_pos={0 0}, upload=454679]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:57362->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:57362->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->45.91.116.9:4261: use of closed network connection
Dec 27 05:43:48.852738396 (518h14m48.723821431s) [noise=true , address=45.91.116.9:4261     , piece_id=YPJ2YX476XJBGTEXMZCY4KRO4P2URNWISNPJDJQW6MNITC2TAU3A, seg_pos={0 0}, upload=454679]: Long tail cancellation due to optimal threshold
Dec 27 05:43:48.852982692 (518h14m48.724065737s) [noise=true , address=45.91.116.13:4155    , piece_id=OLMLVKBSCVLKUMDCXNJ6YHC73D5ZE3WP2JIBPKSFAIIWFQHSGOYA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:41404->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 05:43:48.852990967 (518h14m48.724074002s) [noise=true , address=45.91.116.13:4155    , piece_id=OLMLVKBSCVLKUMDCXNJ6YHC73D5ZE3WP2JIBPKSFAIIWFQHSGOYA, seg_pos={0 0}, upload=454679]: Long tail cancellation due to optimal threshold
Dec 27 05:43:48.853053026 (518h14m48.724136061s) [seg_pos={0 0}, upload=454679]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:43:48.853181971 (518h14m48.724265006s) [upload=454679]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:43:48.853188554 (518h14m48.724271589s) [upload=454679]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:43:49.145082560 (518h14m49.016165605s) [upload=454679]: Done uploading segments...
Dec 27 05:43:49.145106475 (518h14m49.01618952s) [upload=454679]: Upload finished. err: <nil>
Duration: 5.748050289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [.X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......XX]
duration: 
30.003116177s
trace: 
https://jaeger.dev.storj.io/trace/854e50d2395b150
FAIL (2025-12-27 05:44:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 3ORUMBAKKRFYX7LDW3JCFNI5X7PBAWDHLDEVBFXJRAGWLNSDQQRQ) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 2GBYGNMRTA7FGHWAGLAPUMIDR7V2JLWLCJ5YMCMH76QDUFQDRDDA) context deadline exceeded
======LOGS======
alse
Dec 27 05:43:46.233969101 (518h14m46.105052136s) [upload=454679]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:43:46.233974202 (518h14m46.105057237s) [upload=454679]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:43:46.495335648 (518h14m46.366418693s) [seg_pos={0 0}, upload=454679]: Begin upload segment...
Dec 27 05:43:46.495397997 (518h14m46.366481042s) [seg_pos={0 0}, upload=454679]: Done begin upload segment.
Dec 27 05:43:46.495430810 (518h14m46.366513845s) [noise=true , address=145.79.227.38:4979   , piece_id=UUIYYFQ2RHXTV6N6MIRHEGKYNKIYR5G74ZOCJ4HMJGWJG767NMGA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.495518446 (518h14m46.366601481s) [noise=true , address=145.79.227.11:4730   , piece_id=3LPHDP4YQQXCPNTR3HFS3MABDRLC2RIQJTINCQ4LNL3WFR5KQZJQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.496208049 (518h14m46.367291094s) [noise=true , address=145.79.227.51:4410   , piece_id=WOU5YUA75E3NDJMPAPT2N3N322EOHC35ZG2IJHETON7RYHKRZVHQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.496555431 (518h14m46.367638466s) [noise=true , address=145.79.227.46:4602   , piece_id=53V5GAJUSFLQTVDTW7Y4TSL6XS52KNGA5RV32HD5CYPYR2KV2TIQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.496872114 (518h14m46.367955149s) [noise=true , address=45.91.116.7:4009     , piece_id=YFMAMBIXNK6EHCQRSYRUZXJFOWHMOQ7E5TXN2EL23PBKGJM7E26Q, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.497191442 (518h14m46.368274477s) [noise=true , address=145.79.227.42:4123   , piece_id=ITUYKRK53RLGUOWQVMZHAHQCQPKU5V4HBANY7UZNC5SLTGWGAJVA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.497523595 (518h14m46.36860664s) [noise=true , address=45.91.116.9:4261     , piece_id=YPJ2YX476XJBGTEXMZCY4KRO4P2URNWISNPJDJQW6MNITC2TAU3A, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.497888901 (518h14m46.368971936s) [noise=true , address=145.79.227.41:4085   , piece_id=JVMHSEAQLGSIVOTYGGAKTJS4M6NDMAT5U73SPYLPG2YKGBK542PA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.499798165 (518h14m46.3708812s) [noise=true , address=45.91.116.12:4228    , piece_id=2GBYGNMRTA7FGHWAGLAPUMIDR7V2JLWLCJ5YMCMH76QDUFQDRDDA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.501363787 (518h14m46.372446822s) [noise=true , address=77.90.31.17:4773     , piece_id=AMERHRT6KPNHF7UT2XZ27UVG6QVAWLZLCXJ5MJHVSZBKDSIQNH7A, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.503219721 (518h14m46.374302766s) [noise=true , address=45.91.116.10:4084    , piece_id=3ORUMBAKKRFYX7LDW3JCFNI5X7PBAWDHLDEVBFXJRAGWLNSDQQRQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.505097717 (518h14m46.376180752s) [noise=true , address=145.79.227.47:4267   , piece_id=DEHV7R3ZXFGOAEMN5D7YI66523XP4SFNQOWV4HWBSEYNXQAE7ILQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.506464860 (518h14m46.377547895s) [noise=true , address=45.91.116.13:4155    , piece_id=OLMLVKBSCVLKUMDCXNJ6YHC73D5ZE3WP2JIBPKSFAIIWFQHSGOYA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.507770506 (518h14m46.378853541s) [noise=true , address=45.91.116.5:4299     , piece_id=DRYGB3BWR3LYA7KLVQSM5P3CEZF5G3QIKYCTSR23LKC5GFKMPSRQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.509028432 (518h14m46.380111467s) [noise=true , address=145.79.227.12:4765   , piece_id=LQMLIEE7FU65KHQ4BNUV5TRRLWM53ZLHPH6JLJJHZ7VVY4HGGVMQ, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:46.511188966 (518h14m46.382272001s) [noise=true , address=145.79.227.8:4330    , piece_id=L4A6OWPMYFRN4AWNAV7OJJB2CROYUMVHCWWYOTICN45SV4NDTTFA, seg_pos={0 0}, upload=454679]: Uploading piece...
Dec 27 05:43:48.202636992 (518h14m48.073720027s) [upload=454679]: Next returned nil segment
Dec 27 05:43:48.202656579 (518h14m48.073739624s) [upload=454679]: Waiting for error group managing segments...
Dec 27 05:43:48.486541836 (518h14m48.357624871s) [noise=true , address=145.79.227.51:4410   , piece_id=WOU5YUA75E3NDJMPAPT2N3N322EOHC35ZG2IJHETON7RYHKRZVHQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.497739949 (518h14m48.368822984s) [noise=true , address=145.79.227.11:4730   , piece_id=3LPHDP4YQQXCPNTR3HFS3MABDRLC2RIQJTINCQ4LNL3WFR5KQZJQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.507876039 (518h14m48.378959114s) [noise=true , address=145.79.227.12:4765   , piece_id=LQMLIEE7FU65KHQ4BNUV5TRRLWM53ZLHPH6JLJJHZ7VVY4HGGVMQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.510024029 (518h14m48.381107124s) [noise=true , address=145.79.227.42:4123   , piece_id=ITUYKRK53RLGUOWQVMZHAHQCQPKU5V4HBANY7UZNC5SLTGWGAJVA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.516459283 (518h14m48.387542338s) [noise=true , address=145.79.227.8:4330    , piece_id=L4A6OWPMYFRN4AWNAV7OJJB2CROYUMVHCWWYOTICN45SV4NDTTFA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.524420804 (518h14m48.395503839s) [noise=true , address=145.79.227.41:4085   , piece_id=JVMHSEAQLGSIVOTYGGAKTJS4M6NDMAT5U73SPYLPG2YKGBK542PA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.524543968 (518h14m48.395627003s) [noise=true , address=145.79.227.46:4602   , piece_id=53V5GAJUSFLQTVDTW7Y4TSL6XS52KNGA5RV32HD5CYPYR2KV2TIQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.524960290 (518h14m48.396043335s) [noise=true , address=145.79.227.38:4979   , piece_id=UUIYYFQ2RHXTV6N6MIRHEGKYNKIYR5G74ZOCJ4HMJGWJG767NMGA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.550140599 (518h14m48.421223634s) [noise=true , address=145.79.227.47:4267   , piece_id=DEHV7R3ZXFGOAEMN5D7YI66523XP4SFNQOWV4HWBSEYNXQAE7ILQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.734873772 (518h14m48.605956807s) [noise=true , address=45.91.116.7:4009     , piece_id=YFMAMBIXNK6EHCQRSYRUZXJFOWHMOQ7E5TXN2EL23PBKGJM7E26Q, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.800739161 (518h14m48.671822196s) [noise=true , address=45.91.116.10:4084    , piece_id=3ORUMBAKKRFYX7LDW3JCFNI5X7PBAWDHLDEVBFXJRAGWLNSDQQRQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.804571548 (518h14m48.675654593s) [noise=true , address=77.90.31.17:4773     , piece_id=AMERHRT6KPNHF7UT2XZ27UVG6QVAWLZLCXJ5MJHVSZBKDSIQNH7A, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.852468723 (518h14m48.723551768s) [noise=true , address=45.91.116.12:4228    , piece_id=2GBYGNMRTA7FGHWAGLAPUMIDR7V2JLWLCJ5YMCMH76QDUFQDRDDA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: <nil>
Dec 27 05:43:48.852489172 (518h14m48.723572207s) [seg_pos={0 0}, upload=454679]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:43:48.852657062 (518h14m48.723740097s) [noise=true , address=45.91.116.5:4299     , piece_id=DRYGB3BWR3LYA7KLVQSM5P3CEZF5G3QIKYCTSR23LKC5GFKMPSRQ, seg_pos={0 0}, upload=454679]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:54952->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:54952->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54952->45.91.116.5:4299: use of closed network connection
Dec 27 05:43:48.852662632 (518h14m48.723745667s) [noise=true , address=45.91.116.5:4299     , piece_id=DRYGB3BWR3LYA7KLVQSM5P3CEZF5G3QIKYCTSR23LKC5GFKMPSRQ, seg_pos={0 0}, upload=454679]: Long tail cancellation due to optimal threshold
Dec 27 05:43:48.852727356 (518h14m48.723810411s) [noise=true , address=45.91.116.9:4261     , piece_id=YPJ2YX476XJBGTEXMZCY4KRO4P2URNWISNPJDJQW6MNITC2TAU3A, seg_pos={0 0}, upload=454679]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:57362->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:57362->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57362->45.91.116.9:4261: use of closed network connection
Dec 27 05:43:48.852738396 (518h14m48.723821431s) [noise=true , address=45.91.116.9:4261     , piece_id=YPJ2YX476XJBGTEXMZCY4KRO4P2URNWISNPJDJQW6MNITC2TAU3A, seg_pos={0 0}, upload=454679]: Long tail cancellation due to optimal threshold
Dec 27 05:43:48.852982692 (518h14m48.724065737s) [noise=true , address=45.91.116.13:4155    , piece_id=OLMLVKBSCVLKUMDCXNJ6YHC73D5ZE3WP2JIBPKSFAIIWFQHSGOYA, seg_pos={0 0}, upload=454679]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:41404->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 05:43:48.852990967 (518h14m48.724074002s) [noise=true , address=45.91.116.13:4155    , piece_id=OLMLVKBSCVLKUMDCXNJ6YHC73D5ZE3WP2JIBPKSFAIIWFQHSGOYA, seg_pos={0 0}, upload=454679]: Long tail cancellation due to optimal threshold
Dec 27 05:43:48.853053026 (518h14m48.724136061s) [seg_pos={0 0}, upload=454679]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:43:48.853181971 (518h14m48.724265006s) [upload=454679]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:43:48.853188554 (518h14m48.724271589s) [upload=454679]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:43:49.145082560 (518h14m49.016165605s) [upload=454679]: Done uploading segments...
Dec 27 05:43:49.145106475 (518h14m49.01618952s) [upload=454679]: Upload finished. err: <nil>
Duration: 30.00312821s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............................X.............X..............]
duration: 
43.97258124s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:43:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 43.972613231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......X]
duration: 
30.001859619s
trace: 
https://jaeger.dev.storj.io/trace/74ab2fec7e620eed
FAIL (2025-12-27 05:43:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:43:16.008009208 (518h14m15.879092243s) [upload=454671]: Starting upload
Dec 27 05:43:16.008064443 (518h14m15.879147488s) [upload=454671]: Done starting upload
Dec 27 05:43:16.008201635 (518h14m15.87928467s) [upload=454671]: Uploading segments...
Dec 27 05:43:16.008309279 (518h14m15.879392325s) [upload=454671]: Got next segment. Inline: false
Dec 27 05:43:16.008351430 (518h14m15.879434465s) [upload=454671]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:43:16.008361038 (518h14m15.879444073s) [upload=454671]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:43:16.250235879 (518h14m16.121318934s) [seg_pos={0 0}, upload=454671]: Begin upload segment...
Dec 27 05:43:16.250286134 (518h14m16.121369179s) [seg_pos={0 0}, upload=454671]: Done begin upload segment.
Dec 27 05:43:16.250314488 (518h14m16.121397523s) [noise=true , address=45.91.116.7:4009     , piece_id=G7TJXIUW7MMLNJ3K7HRFDNXJMW5LR7QWMKFZHNFI2VKZ4KFNQXOQ, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.250333695 (518h14m16.12141674s) [noise=true , address=145.79.227.45:4230   , piece_id=EKK4FEF4FQUJ7GFGNKQ76BUHHZ6ZKSECPZDL6ZWZGHA4CXKHTOZQ, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.250479693 (518h14m16.121562738s) [noise=true , address=145.79.227.17:4971   , piece_id=B4BHVU7MO3FQKHZEAKD2RFPDOCUQH5BGK4RXCMX23MBPY3PP6SEA, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.250957142 (518h14m16.122040187s) [noise=true , address=45.91.116.13:4155    , piece_id=33TUYFB5VS6SNRLLBMSPC7YRIVPZO54XY2E47FZUSGHYBYUEZKUA, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.251469668 (518h14m16.122552703s) [noise=true , address=45.91.116.5:4298     , piece_id=K7NQS62Y7IFQRRH76YDVVWAV43N7I7CMWDQ2L7CQ5347PU3N4A7A, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.251918342 (518h14m16.123001387s) [noise=true , address=145.79.227.36:4518   , piece_id=EXTUTYFPBUIVGKBSY6LGFB4UMUHURV5ZNNIGF7EGIAOMXYYGKSBQ, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.252338993 (518h14m16.123422038s) [noise=true , address=77.90.31.10:4517     , piece_id=WS3677C42M3WQBQBS2K53Q4TTLBZIJBVF5WQZZLKTLAQLVSB6OAQ, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.253127104 (518h14m16.124210149s) [noise=true , address=145.79.227.39:4013   , piece_id=GB2A4YDIHYJCKVCFTFUCBV566JQVVZKMNQPKWSXX45JXMIHFETEA, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.253866061 (518h14m16.124949106s) [noise=true , address=45.91.116.6:4973     , piece_id=KRGB3YVKFF4JKP3X3ABIZ4XIMPJLNLX7V6OJ2AAOUY2MHFQQQPBQ, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.254695370 (518h14m16.125778405s) [noise=true , address=145.79.227.41:4085   , piece_id=QTWMXP4LZBAISX3DJJLJLRCF54LE55KDBWZ3TPXJKVZ2CUQU46DA, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.255869667 (518h14m16.126952702s) [noise=true , address=145.79.227.48:4303   , piece_id=GT5QE6LH2G6THY52TRRCBR3UJ7F7PUASHL26RISB65ZPYSNOHE7Q, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.257116962 (518h14m16.128200007s) [noise=true , address=45.91.116.8:4048     , piece_id=FOWZJK37DQOFN3M4LWPLF2FSI2Y5KFZ3LRVFJQN63CS5RXGZSXCQ, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.258391799 (518h14m16.129474834s) [noise=true , address=145.79.227.31:4798   , piece_id=HPLOZC3Z4OBMQ2OUMHMICYC5DBWSRVYNDC33ZAGHWVUKGLUDN64A, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.259668861 (518h14m16.130751906s) [noise=true , address=145.79.227.37:4567   , piece_id=EPAL46YTVORK7BCSSKVUVGIYZTAUXJOJCUSOT5U4LFO2Q6462XYA, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.262429898 (518h14m16.133512933s) [noise=true , address=145.79.227.38:4977   , piece_id=BQVRAR2EN45FLPTAJSLCPDYE44DGXODHQ7CFRSV5FHNIIXGGDSSQ, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:16.262961551 (518h14m16.134044596s) [noise=true , address=145.79.227.51:4411   , piece_id=TYMM7DOUYDF34DQFNSPOFNZ7A5FRJQWKZGF2EPB7KTXBTZH3ZORA, seg_pos={0 0}, upload=454671]: Uploading piece...
Dec 27 05:43:17.968554902 (518h14m17.839637947s) [upload=454671]: Next returned nil segment
Dec 27 05:43:17.968597964 (518h14m17.839681029s) [upload=454671]: Waiting for error group managing segments...
Dec 27 05:43:18.258327278 (518h14m18.129410323s) [noise=true , address=145.79.227.41:4085   , piece_id=QTWMXP4LZBAISX3DJJLJLRCF54LE55KDBWZ3TPXJKVZ2CUQU46DA, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.269136260 (518h14m18.140219295s) [noise=true , address=145.79.227.51:4411   , piece_id=TYMM7DOUYDF34DQFNSPOFNZ7A5FRJQWKZGF2EPB7KTXBTZH3ZORA, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.270253588 (518h14m18.141336623s) [noise=true , address=145.79.227.31:4798   , piece_id=HPLOZC3Z4OBMQ2OUMHMICYC5DBWSRVYNDC33ZAGHWVUKGLUDN64A, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.274562232 (518h14m18.145645267s) [noise=true , address=145.79.227.36:4518   , piece_id=EXTUTYFPBUIVGKBSY6LGFB4UMUHURV5ZNNIGF7EGIAOMXYYGKSBQ, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.277384216 (518h14m18.148467251s) [noise=true , address=145.79.227.17:4971   , piece_id=B4BHVU7MO3FQKHZEAKD2RFPDOCUQH5BGK4RXCMX23MBPY3PP6SEA, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.279980699 (518h14m18.151063744s) [noise=true , address=145.79.227.48:4303   , piece_id=GT5QE6LH2G6THY52TRRCBR3UJ7F7PUASHL26RISB65ZPYSNOHE7Q, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.280255122 (518h14m18.151338167s) [noise=true , address=145.79.227.45:4230   , piece_id=EKK4FEF4FQUJ7GFGNKQ76BUHHZ6ZKSECPZDL6ZWZGHA4CXKHTOZQ, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.283526050 (518h14m18.154609085s) [noise=true , address=145.79.227.38:4977   , piece_id=BQVRAR2EN45FLPTAJSLCPDYE44DGXODHQ7CFRSV5FHNIIXGGDSSQ, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.286138856 (518h14m18.157221911s) [noise=true , address=145.79.227.39:4013   , piece_id=GB2A4YDIHYJCKVCFTFUCBV566JQVVZKMNQPKWSXX45JXMIHFETEA, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.296905107 (518h14m18.167988142s) [noise=true , address=145.79.227.37:4567   , piece_id=EPAL46YTVORK7BCSSKVUVGIYZTAUXJOJCUSOT5U4LFO2Q6462XYA, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.507935114 (518h14m18.379018159s) [noise=true , address=45.91.116.13:4155    , piece_id=33TUYFB5VS6SNRLLBMSPC7YRIVPZO54XY2E47FZUSGHYBYUEZKUA, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.557483212 (518h14m18.428566277s) [noise=true , address=45.91.116.6:4973     , piece_id=KRGB3YVKFF4JKP3X3ABIZ4XIMPJLNLX7V6OJ2AAOUY2MHFQQQPBQ, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.563167035 (518h14m18.43425008s) [noise=true , address=77.90.31.10:4517     , piece_id=WS3677C42M3WQBQBS2K53Q4TTLBZIJBVF5WQZZLKTLAQLVSB6OAQ, seg_pos={0 0}, upload=454671]: Done uploading piece. err: <nil>
Dec 27 05:43:18.563194818 (518h14m18.434277853s) [seg_pos={0 0}, upload=454671]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:43:18.563450975 (518h14m18.434534041s) [noise=true , address=45.91.116.7:4009     , piece_id=G7TJXIUW7MMLNJ3K7HRFDNXJMW5LR7QWMKFZHNFI2VKZ4KFNQXOQ, seg_pos={0 0}, upload=454671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 05:43:18.563461265 (518h14m18.43454431s) [noise=true , address=45.91.116.7:4009     , piece_id=G7TJXIUW7MMLNJ3K7HRFDNXJMW5LR7QWMKFZHNFI2VKZ4KFNQXOQ, seg_pos={0 0}, upload=454671]: Long tail cancellation due to optimal threshold
Dec 27 05:43:18.563597504 (518h14m18.434680539s) [noise=true , address=45.91.116.8:4048     , piece_id=FOWZJK37DQOFN3M4LWPLF2FSI2Y5KFZ3LRVFJQN63CS5RXGZSXCQ, seg_pos={0 0}, upload=454671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 27 05:43:18.563613615 (518h14m18.43469666s) [noise=true , address=45.91.116.8:4048     , piece_id=FOWZJK37DQOFN3M4LWPLF2FSI2Y5KFZ3LRVFJQN63CS5RXGZSXCQ, seg_pos={0 0}, upload=454671]: Long tail cancellation due to optimal threshold
Dec 27 05:43:18.563749344 (518h14m18.434832379s) [noise=true , address=45.91.116.5:4298     , piece_id=K7NQS62Y7IFQRRH76YDVVWAV43N7I7CMWDQ2L7CQ5347PU3N4A7A, seg_pos={0 0}, upload=454671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 27 05:43:18.563761978 (518h14m18.434845023s) [noise=true , address=45.91.116.5:4298     , piece_id=K7NQS62Y7IFQRRH76YDVVWAV43N7I7CMWDQ2L7CQ5347PU3N4A7A, seg_pos={0 0}, upload=454671]: Long tail cancellation due to optimal threshold
Dec 27 05:43:18.563783168 (518h14m18.434866203s) [seg_pos={0 0}, upload=454671]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:43:18.563884361 (518h14m18.434967396s) [upload=454671]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:43:18.563889351 (518h14m18.434972396s) [upload=454671]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:43:18.851451177 (518h14m18.722534212s) [upload=454671]: Done uploading segments...
Dec 27 05:43:18.851470625 (518h14m18.72255366s) [upload=454671]: Upload finished. err: <nil>
Duration: 30.001885448s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.............X..............................................]
duration: 
3.580132805s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:43:43 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.580168794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..............X.............................................]
duration: 
3.520405523s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:43:22 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.520434378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...............X............................................]
duration: 
3.561326033s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:43:04 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.561344598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [................................X.............X.............]
duration: 
40.164812782s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:42:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 40.164838942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [X..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......]
duration: 
5.874291094s
trace: 
https://jaeger.dev.storj.io/trace/48e8aa9553656273
FAIL (2025-12-27 05:42:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded
======LOGS======
Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload
Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload
Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments...
Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false
Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment...
Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment.
Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment
Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments...
Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments...
Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil>
Duration: 5.874329718s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [................X...........................................]
duration: 
3.587914753s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:41:54 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.587951504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.................................X.............X............]
duration: 
36.213006746s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:41:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 36.213035531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [XX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX.....]
duration: 
6.110578365s
trace: 
https://jaeger.dev.storj.io/trace/48e8aa9553656273
FAIL (2025-12-27 05:41:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded
======LOGS======
Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload
Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload
Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments...
Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false
Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment...
Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment.
Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment
Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments...
Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments...
Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil>
Duration: 6.110609334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [XXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX....]
duration: 
15.016613941s
trace: 
https://jaeger.dev.storj.io/trace/48e8aa9553656273
FAIL (2025-12-27 05:41:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded
======LOGS======
Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload
Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload
Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments...
Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false
Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment...
Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment.
Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment
Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments...
Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments...
Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil>
Duration: 15.016643487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..................................X.............X...........]
duration: 
34.000006612s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:41:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 34.000021661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.................X..........................................]
duration: 
3.484115119s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:40:45 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.48414123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [X..................................X.............X..........]
duration: 
38.882465477s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:40:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 38.882493701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX...]
duration: 
6.252727001s
trace: 
https://jaeger.dev.storj.io/trace/48e8aa9553656273
FAIL (2025-12-27 05:40:27 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded
======LOGS======
Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload
Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload
Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments...
Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false
Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment...
Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment.
Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment
Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments...
Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments...
Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil>
Duration: 6.252755626s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [.XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX..]
duration: 
5.753114873s
trace: 
https://jaeger.dev.storj.io/trace/48e8aa9553656273
FAIL (2025-12-27 05:39:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded
======LOGS======
Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload
Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload
Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments...
Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false
Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment...
Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment.
Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment
Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments...
Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments...
Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil>
Duration: 5.753141834s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..................X.........................................]
duration: 
3.403695136s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:39:51 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.40372357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.X..................................X.............X.........]
duration: 
30.409790845s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:39:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 30.409811515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..X..................................X.............X........]
duration: 
37.037788088s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:39:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 37.03780448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [..XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX.]
duration: 
5.765558538s
trace: 
https://jaeger.dev.storj.io/trace/48e8aa9553656273
FAIL (2025-12-27 05:39:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded
======LOGS======
Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload
Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload
Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments...
Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false
Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment...
Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment.
Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment
Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments...
Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments...
Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil>
Duration: 5.765576241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [...XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX]
duration: 
30.003245589s
trace: 
https://jaeger.dev.storj.io/trace/48e8aa9553656273
FAIL (2025-12-27 05:39:05 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded
======LOGS======
Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload
Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload
Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments...
Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false
Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment...
Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment.
Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece...
Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment
Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments...
Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302   , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798   , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765   , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229   , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014   , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122   , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267   , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873   , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977   , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447   , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519     , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118    , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263     , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil>
Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808     , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156    , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878     , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold
Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments...
Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil>
Duration: 30.003257372s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...................X........................................]
duration: 
3.553004748s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:38:50 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.553030796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...X..................................X.............X.......]
duration: 
41.492355808s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:38:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 41.492386175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...X]
duration: 
30.002673568s
trace: 
https://jaeger.dev.storj.io/trace/3749de954066bfbb
FAIL (2025-12-27 05:38:35 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:38:05.703447414 (518h9m5.57453046s) [upload=454595]: Starting upload
Dec 27 05:38:05.703548487 (518h9m5.574631532s) [upload=454595]: Done starting upload
Dec 27 05:38:05.703597411 (518h9m5.574680466s) [upload=454595]: Uploading segments...
Dec 27 05:38:05.703835654 (518h9m5.574918699s) [upload=454595]: Got next segment. Inline: false
Dec 27 05:38:05.703878837 (518h9m5.574961882s) [upload=454595]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:38:05.703885018 (518h9m5.574968063s) [upload=454595]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:38:05.956331880 (518h9m5.827414925s) [seg_pos={0 0}, upload=454595]: Begin upload segment...
Dec 27 05:38:05.956389159 (518h9m5.827472194s) [seg_pos={0 0}, upload=454595]: Done begin upload segment.
Dec 27 05:38:05.956421671 (518h9m5.827504706s) [noise=true , address=145.79.227.42:4121   , piece_id=4WNRFOLHFPFPY6AXZC26KABO3A6ZLRDXMZ5OITARXEUS3P2CSJOA, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.956432692 (518h9m5.827515777s) [noise=true , address=145.79.227.49:4339   , piece_id=NNM5ZHZYZF6MHNWPXLB54KV7TSSANB7AAWVNWE3HFD7A4TOQW46A, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.956681916 (518h9m5.827764961s) [noise=true , address=145.79.227.16:4911   , piece_id=PWJTZHKKLOCLNQYEBQA3BXTMQE2C7XWTLPA7LFKHW2EGT3TUWSIQ, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.957270086 (518h9m5.828353121s) [noise=true , address=145.79.227.41:4087   , piece_id=4UMW3L3HEVGWTWSA3FGCZXPMPYKPBK2EPDOBQB7KQSWPF27EFP5Q, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.957596297 (518h9m5.828679342s) [noise=true , address=145.79.227.12:4765   , piece_id=2ZCUGTBIHHA2LFYJG2FOCQ4WOOOCHD5WAYIFLJ3ASXRMR37BATSQ, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.958055140 (518h9m5.829138176s) [noise=true , address=145.79.227.37:4566   , piece_id=LFOMJWVLCQZWBDLMOKOKHPXRJMOLPEJCL4ALCK5D3GXRQTCCOCOQ, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.958407011 (518h9m5.829490056s) [noise=true , address=77.90.31.19:4841     , piece_id=GE3FNE5OTBBSY3GCYY5LN52T46WA3IDIOF2VLJVYKOGKJATO5E7A, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.958735286 (518h9m5.829818321s) [noise=true , address=45.91.116.9:4261     , piece_id=IJRKJSZMRIXALRA6R4MI65YYVAVOBQXMQ5KCQUUCA25WSC7A5TYA, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.959194390 (518h9m5.830277485s) [noise=true , address=45.91.116.4:4937     , piece_id=2WYWBMHLXKEG7X7BWRU6W3KPLG25BLAJHRXOXWT4DB2Y7HY7P5MA, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.959545358 (518h9m5.830628403s) [noise=true , address=145.79.227.39:4015   , piece_id=7HDU27GQTZRD2YPAXQ7GAUWJMDF5KI5UBTE6MJB23ETJEBRHI5NA, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.959697929 (518h9m5.830780975s) [noise=true , address=45.91.116.13:4155    , piece_id=JMBJKL5EWIYDGNHGHEK5ZDPUTW6PES64RPOCGYHO6Y6L3KHLJTIA, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.960014693 (518h9m5.831097728s) [noise=true , address=145.79.227.26:4654   , piece_id=JBKE7EQ4QJBFSRLZW2SNO5BQPE4ONRNZHAB3BQM4XNQ3MLV5U2CQ, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.960316026 (518h9m5.831399061s) [noise=true , address=145.79.227.48:4303   , piece_id=6YVDDD3A6QPVYNTQ4U3GGMFZDEYRJW3WNJZS6HJYXJSPP6EWCQ2Q, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.960656354 (518h9m5.831739389s) [noise=true , address=145.79.227.28:4725   , piece_id=23X5AKKFKS6ZIONIHKYEXEHV64WLQVJILKL6D7QULBU2TYB2MCWA, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.960691090 (518h9m5.831774155s) [noise=true , address=145.79.227.2:4113    , piece_id=Y3TEXMHMU3T6TJTZ65NPA52BYSB4DYUKY3SXCC4SWJWTUXRZ3TZQ, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:05.960992574 (518h9m5.83207562s) [noise=true , address=145.79.227.47:4266   , piece_id=HOGSCKCVNHO45LI6CLVMCWQDD3LYF3EB6GWNY5WQCEKADF6ERQQA, seg_pos={0 0}, upload=454595]: Uploading piece...
Dec 27 05:38:07.569690160 (518h9m7.440773195s) [upload=454595]: Next returned nil segment
Dec 27 05:38:07.569703725 (518h9m7.44078676s) [upload=454595]: Waiting for error group managing segments...
Dec 27 05:38:07.955580530 (518h9m7.826663575s) [noise=true , address=145.79.227.39:4015   , piece_id=7HDU27GQTZRD2YPAXQ7GAUWJMDF5KI5UBTE6MJB23ETJEBRHI5NA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:07.958978952 (518h9m7.830061987s) [noise=true , address=145.79.227.12:4765   , piece_id=2ZCUGTBIHHA2LFYJG2FOCQ4WOOOCHD5WAYIFLJ3ASXRMR37BATSQ, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:07.965378548 (518h9m7.836461593s) [noise=true , address=145.79.227.41:4087   , piece_id=4UMW3L3HEVGWTWSA3FGCZXPMPYKPBK2EPDOBQB7KQSWPF27EFP5Q, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:07.966018997 (518h9m7.837102032s) [noise=true , address=145.79.227.48:4303   , piece_id=6YVDDD3A6QPVYNTQ4U3GGMFZDEYRJW3WNJZS6HJYXJSPP6EWCQ2Q, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:07.966844770 (518h9m7.837927805s) [noise=true , address=145.79.227.28:4725   , piece_id=23X5AKKFKS6ZIONIHKYEXEHV64WLQVJILKL6D7QULBU2TYB2MCWA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:07.974160039 (518h9m7.845243084s) [noise=true , address=145.79.227.26:4654   , piece_id=JBKE7EQ4QJBFSRLZW2SNO5BQPE4ONRNZHAB3BQM4XNQ3MLV5U2CQ, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:07.994572978 (518h9m7.865656014s) [noise=true , address=145.79.227.42:4121   , piece_id=4WNRFOLHFPFPY6AXZC26KABO3A6ZLRDXMZ5OITARXEUS3P2CSJOA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:07.995578794 (518h9m7.866661839s) [noise=true , address=145.79.227.37:4566   , piece_id=LFOMJWVLCQZWBDLMOKOKHPXRJMOLPEJCL4ALCK5D3GXRQTCCOCOQ, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:07.996024162 (518h9m7.867107197s) [noise=true , address=145.79.227.16:4911   , piece_id=PWJTZHKKLOCLNQYEBQA3BXTMQE2C7XWTLPA7LFKHW2EGT3TUWSIQ, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:07.996478637 (518h9m7.867561672s) [noise=true , address=145.79.227.2:4113    , piece_id=Y3TEXMHMU3T6TJTZ65NPA52BYSB4DYUKY3SXCC4SWJWTUXRZ3TZQ, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:07.996834214 (518h9m7.867917249s) [noise=true , address=145.79.227.47:4266   , piece_id=HOGSCKCVNHO45LI6CLVMCWQDD3LYF3EB6GWNY5WQCEKADF6ERQQA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:08.000784727 (518h9m7.871867762s) [noise=true , address=145.79.227.49:4339   , piece_id=NNM5ZHZYZF6MHNWPXLB54KV7TSSANB7AAWVNWE3HFD7A4TOQW46A, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:08.197039069 (518h9m8.068122114s) [noise=true , address=45.91.116.13:4155    , piece_id=JMBJKL5EWIYDGNHGHEK5ZDPUTW6PES64RPOCGYHO6Y6L3KHLJTIA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:08.197063465 (518h9m8.06814651s) [seg_pos={0 0}, upload=454595]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:38:08.197356182 (518h9m8.068439227s) [noise=true , address=45.91.116.4:4937     , piece_id=2WYWBMHLXKEG7X7BWRU6W3KPLG25BLAJHRXOXWT4DB2Y7HY7P5MA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:08.197365490 (518h9m8.068448535s) [noise=true , address=45.91.116.4:4937     , piece_id=2WYWBMHLXKEG7X7BWRU6W3KPLG25BLAJHRXOXWT4DB2Y7HY7P5MA, seg_pos={0 0}, upload=454595]: Long tail cancellation due to optimal threshold
Dec 27 05:38:08.197781532 (518h9m8.068864577s) [noise=true , address=45.91.116.9:4261     , piece_id=IJRKJSZMRIXALRA6R4MI65YYVAVOBQXMQ5KCQUUCA25WSC7A5TYA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 05:38:08.197793295 (518h9m8.06887634s) [noise=true , address=45.91.116.9:4261     , piece_id=IJRKJSZMRIXALRA6R4MI65YYVAVOBQXMQ5KCQUUCA25WSC7A5TYA, seg_pos={0 0}, upload=454595]: Long tail cancellation due to optimal threshold
Dec 27 05:38:08.198939889 (518h9m8.070022924s) [noise=true , address=77.90.31.19:4841     , piece_id=GE3FNE5OTBBSY3GCYY5LN52T46WA3IDIOF2VLJVYKOGKJATO5E7A, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil>
Dec 27 05:38:08.198987489 (518h9m8.070070524s) [seg_pos={0 0}, upload=454595]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 27 05:38:08.199114822 (518h9m8.070197877s) [upload=454595]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:38:08.199122797 (518h9m8.070205832s) [upload=454595]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:38:08.488019957 (518h9m8.359102992s) [upload=454595]: Done uploading segments...
Dec 27 05:38:08.488041948 (518h9m8.359124993s) [upload=454595]: Upload finished. err: <nil>
Duration: 30.002699839s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [....................X.......................................]
duration: 
3.483796525s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:38:23 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.483824547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.....................X......................................]
duration: 
3.592469886s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:38:05 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.592497888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [....X..................................X.............X......]
duration: 
35.426153454s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:37:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 35.426168684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
27 out of 60 is failed (threshold=3) [X....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...]
duration: 
5.934749357s
trace: 
https://jaeger.dev.storj.io/trace/4170035c95ee3ad0
FAIL (2025-12-27 05:37:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:34:48.380482691 (518h5m48.251565726s) [upload=454544]: Starting upload
Dec 27 05:34:48.380562924 (518h5m48.25164596s) [upload=454544]: Done starting upload
Dec 27 05:34:48.380673735 (518h5m48.25175678s) [upload=454544]: Uploading segments...
Dec 27 05:34:48.380782292 (518h5m48.251865337s) [upload=454544]: Got next segment. Inline: false
Dec 27 05:34:48.380815305 (518h5m48.25189835s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:34:48.380820384 (518h5m48.251903419s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:34:48.626473509 (518h5m48.497556555s) [seg_pos={0 0}, upload=454544]: Begin upload segment...
Dec 27 05:34:48.626528334 (518h5m48.497611369s) [seg_pos={0 0}, upload=454544]: Done begin upload segment.
Dec 27 05:34:48.626556177 (518h5m48.497639212s) [noise=true , address=145.79.227.40:4050   , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.628331186 (518h5m48.499414232s) [noise=true , address=145.79.227.5:4221    , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.628728574 (518h5m48.499811619s) [noise=true , address=145.79.227.49:4338   , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629099119 (518h5m48.500182154s) [noise=true , address=145.79.227.41:4086   , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629419209 (518h5m48.500502244s) [noise=true , address=145.79.227.9:4366    , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629760218 (518h5m48.500843253s) [noise=true , address=45.91.116.6:4973     , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.630136835 (518h5m48.50121988s) [noise=true , address=145.79.227.37:4566   , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.631218024 (518h5m48.502301069s) [noise=true , address=145.79.227.42:4122   , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.632043446 (518h5m48.503126491s) [noise=true , address=145.79.227.48:4303   , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.632670550 (518h5m48.503753585s) [noise=true , address=45.91.116.4:4940     , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.634500525 (518h5m48.50558357s) [noise=true , address=145.79.227.2:4114    , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.635139160 (518h5m48.506222206s) [noise=true , address=45.91.116.13:4154    , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.635823263 (518h5m48.506906309s) [noise=true , address=145.79.227.36:4518   , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.636522755 (518h5m48.50760579s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.637098070 (518h5m48.508181105s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.637850353 (518h5m48.508933388s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:50.260369707 (518h5m50.131452752s) [upload=454544]: Next returned nil segment
Dec 27 05:34:50.260382743 (518h5m50.131465778s) [upload=454544]: Waiting for error group managing segments...
Dec 27 05:34:50.639212409 (518h5m50.510295444s) [noise=true , address=145.79.227.5:4221    , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.640973903 (518h5m50.512056938s) [noise=true , address=145.79.227.36:4518   , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.641271821 (518h5m50.512354866s) [noise=true , address=145.79.227.9:4366    , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.641696529 (518h5m50.512779564s) [noise=true , address=145.79.227.41:4086   , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.642399538 (518h5m50.513482583s) [noise=true , address=145.79.227.2:4114    , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.644172935 (518h5m50.51525597s) [noise=true , address=145.79.227.49:4338   , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.651155731 (518h5m50.522238776s) [noise=true , address=145.79.227.40:4050   , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.656354861 (518h5m50.527437896s) [noise=true , address=145.79.227.48:4303   , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.659760366 (518h5m50.530843411s) [noise=true , address=145.79.227.37:4566   , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.672522708 (518h5m50.543605743s) [noise=true , address=145.79.227.42:4122   , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.908330215 (518h5m50.77941326s) [noise=true , address=45.91.116.13:4154    , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.000300392 (518h5m50.871383427s) [noise=true , address=45.91.116.4:4940     , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.012693621 (518h5m50.883776676s) [noise=true , address=45.91.116.6:4973     , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.012717165 (518h5m50.88380021s) [seg_pos={0 0}, upload=454544]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:34:51.012957062 (518h5m50.884040107s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.012968935 (518h5m50.884051981s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013095285 (518h5m50.884178331s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.013113350 (518h5m50.884196395s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013296098 (518h5m50.884379143s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.013303251 (518h5m50.884386287s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013325995 (518h5m50.88440903s) [seg_pos={0 0}, upload=454544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:34:51.013423311 (518h5m50.884506346s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:34:51.013492492 (518h5m50.884575537s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:34:51.301485099 (518h5m51.172568134s) [upload=454544]: Done uploading segments...
Dec 27 05:34:51.301504687 (518h5m51.172587732s) [upload=454544]: Upload finished. err: <nil>
Duration: 5.934779595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.....X..................................X.............X.....]
duration: 
48.607477528s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:37:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 48.607494841s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [......................X.....................................]
duration: 
3.598799941s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:36:59 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.598867119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX..]
duration: 
6.11886824s
trace: 
https://jaeger.dev.storj.io/trace/4170035c95ee3ad0
FAIL (2025-12-27 05:36:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:34:48.380482691 (518h5m48.251565726s) [upload=454544]: Starting upload
Dec 27 05:34:48.380562924 (518h5m48.25164596s) [upload=454544]: Done starting upload
Dec 27 05:34:48.380673735 (518h5m48.25175678s) [upload=454544]: Uploading segments...
Dec 27 05:34:48.380782292 (518h5m48.251865337s) [upload=454544]: Got next segment. Inline: false
Dec 27 05:34:48.380815305 (518h5m48.25189835s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:34:48.380820384 (518h5m48.251903419s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:34:48.626473509 (518h5m48.497556555s) [seg_pos={0 0}, upload=454544]: Begin upload segment...
Dec 27 05:34:48.626528334 (518h5m48.497611369s) [seg_pos={0 0}, upload=454544]: Done begin upload segment.
Dec 27 05:34:48.626556177 (518h5m48.497639212s) [noise=true , address=145.79.227.40:4050   , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.628331186 (518h5m48.499414232s) [noise=true , address=145.79.227.5:4221    , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.628728574 (518h5m48.499811619s) [noise=true , address=145.79.227.49:4338   , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629099119 (518h5m48.500182154s) [noise=true , address=145.79.227.41:4086   , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629419209 (518h5m48.500502244s) [noise=true , address=145.79.227.9:4366    , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629760218 (518h5m48.500843253s) [noise=true , address=45.91.116.6:4973     , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.630136835 (518h5m48.50121988s) [noise=true , address=145.79.227.37:4566   , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.631218024 (518h5m48.502301069s) [noise=true , address=145.79.227.42:4122   , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.632043446 (518h5m48.503126491s) [noise=true , address=145.79.227.48:4303   , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.632670550 (518h5m48.503753585s) [noise=true , address=45.91.116.4:4940     , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.634500525 (518h5m48.50558357s) [noise=true , address=145.79.227.2:4114    , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.635139160 (518h5m48.506222206s) [noise=true , address=45.91.116.13:4154    , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.635823263 (518h5m48.506906309s) [noise=true , address=145.79.227.36:4518   , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.636522755 (518h5m48.50760579s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.637098070 (518h5m48.508181105s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.637850353 (518h5m48.508933388s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:50.260369707 (518h5m50.131452752s) [upload=454544]: Next returned nil segment
Dec 27 05:34:50.260382743 (518h5m50.131465778s) [upload=454544]: Waiting for error group managing segments...
Dec 27 05:34:50.639212409 (518h5m50.510295444s) [noise=true , address=145.79.227.5:4221    , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.640973903 (518h5m50.512056938s) [noise=true , address=145.79.227.36:4518   , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.641271821 (518h5m50.512354866s) [noise=true , address=145.79.227.9:4366    , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.641696529 (518h5m50.512779564s) [noise=true , address=145.79.227.41:4086   , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.642399538 (518h5m50.513482583s) [noise=true , address=145.79.227.2:4114    , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.644172935 (518h5m50.51525597s) [noise=true , address=145.79.227.49:4338   , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.651155731 (518h5m50.522238776s) [noise=true , address=145.79.227.40:4050   , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.656354861 (518h5m50.527437896s) [noise=true , address=145.79.227.48:4303   , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.659760366 (518h5m50.530843411s) [noise=true , address=145.79.227.37:4566   , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.672522708 (518h5m50.543605743s) [noise=true , address=145.79.227.42:4122   , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.908330215 (518h5m50.77941326s) [noise=true , address=45.91.116.13:4154    , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.000300392 (518h5m50.871383427s) [noise=true , address=45.91.116.4:4940     , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.012693621 (518h5m50.883776676s) [noise=true , address=45.91.116.6:4973     , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.012717165 (518h5m50.88380021s) [seg_pos={0 0}, upload=454544]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:34:51.012957062 (518h5m50.884040107s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.012968935 (518h5m50.884051981s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013095285 (518h5m50.884178331s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.013113350 (518h5m50.884196395s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013296098 (518h5m50.884379143s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.013303251 (518h5m50.884386287s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013325995 (518h5m50.88440903s) [seg_pos={0 0}, upload=454544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:34:51.013423311 (518h5m50.884506346s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:34:51.013492492 (518h5m50.884575537s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:34:51.301485099 (518h5m51.172568134s) [upload=454544]: Done uploading segments...
Dec 27 05:34:51.301504687 (518h5m51.172587732s) [upload=454544]: Upload finished. err: <nil>
Duration: 6.118897868s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.......................X....................................]
duration: 
3.688536384s
trace: 
https://jaeger.dev.storj.io/trace/bb932b4ff37ef93
FAIL (2025-12-27 05:36:02 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.688563224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [......X..................................X.............X....]
duration: 
46.644974513s
trace: 
https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06
FAIL (2025-12-27 05:35:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece...
Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment
Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments...
Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321   , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171   , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454   , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832     , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633     , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099    , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767     , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066     , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849     , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596     , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173    , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219   , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928   , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802     , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil>
Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces.
Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection
Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715    , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064   , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778   , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection
Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549     , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold
Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments...
Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil>
Duration: 46.644994542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
28 out of 60 is failed (threshold=3) [.XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX.]
duration: 
5.811565903s
trace: 
https://jaeger.dev.storj.io/trace/4170035c95ee3ad0
FAIL (2025-12-27 05:35:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:34:48.380482691 (518h5m48.251565726s) [upload=454544]: Starting upload
Dec 27 05:34:48.380562924 (518h5m48.25164596s) [upload=454544]: Done starting upload
Dec 27 05:34:48.380673735 (518h5m48.25175678s) [upload=454544]: Uploading segments...
Dec 27 05:34:48.380782292 (518h5m48.251865337s) [upload=454544]: Got next segment. Inline: false
Dec 27 05:34:48.380815305 (518h5m48.25189835s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:34:48.380820384 (518h5m48.251903419s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:34:48.626473509 (518h5m48.497556555s) [seg_pos={0 0}, upload=454544]: Begin upload segment...
Dec 27 05:34:48.626528334 (518h5m48.497611369s) [seg_pos={0 0}, upload=454544]: Done begin upload segment.
Dec 27 05:34:48.626556177 (518h5m48.497639212s) [noise=true , address=145.79.227.40:4050   , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.628331186 (518h5m48.499414232s) [noise=true , address=145.79.227.5:4221    , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.628728574 (518h5m48.499811619s) [noise=true , address=145.79.227.49:4338   , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629099119 (518h5m48.500182154s) [noise=true , address=145.79.227.41:4086   , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629419209 (518h5m48.500502244s) [noise=true , address=145.79.227.9:4366    , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629760218 (518h5m48.500843253s) [noise=true , address=45.91.116.6:4973     , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.630136835 (518h5m48.50121988s) [noise=true , address=145.79.227.37:4566   , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.631218024 (518h5m48.502301069s) [noise=true , address=145.79.227.42:4122   , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.632043446 (518h5m48.503126491s) [noise=true , address=145.79.227.48:4303   , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.632670550 (518h5m48.503753585s) [noise=true , address=45.91.116.4:4940     , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.634500525 (518h5m48.50558357s) [noise=true , address=145.79.227.2:4114    , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.635139160 (518h5m48.506222206s) [noise=true , address=45.91.116.13:4154    , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.635823263 (518h5m48.506906309s) [noise=true , address=145.79.227.36:4518   , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.636522755 (518h5m48.50760579s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.637098070 (518h5m48.508181105s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.637850353 (518h5m48.508933388s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:50.260369707 (518h5m50.131452752s) [upload=454544]: Next returned nil segment
Dec 27 05:34:50.260382743 (518h5m50.131465778s) [upload=454544]: Waiting for error group managing segments...
Dec 27 05:34:50.639212409 (518h5m50.510295444s) [noise=true , address=145.79.227.5:4221    , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.640973903 (518h5m50.512056938s) [noise=true , address=145.79.227.36:4518   , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.641271821 (518h5m50.512354866s) [noise=true , address=145.79.227.9:4366    , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.641696529 (518h5m50.512779564s) [noise=true , address=145.79.227.41:4086   , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.642399538 (518h5m50.513482583s) [noise=true , address=145.79.227.2:4114    , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.644172935 (518h5m50.51525597s) [noise=true , address=145.79.227.49:4338   , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.651155731 (518h5m50.522238776s) [noise=true , address=145.79.227.40:4050   , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.656354861 (518h5m50.527437896s) [noise=true , address=145.79.227.48:4303   , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.659760366 (518h5m50.530843411s) [noise=true , address=145.79.227.37:4566   , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.672522708 (518h5m50.543605743s) [noise=true , address=145.79.227.42:4122   , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.908330215 (518h5m50.77941326s) [noise=true , address=45.91.116.13:4154    , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.000300392 (518h5m50.871383427s) [noise=true , address=45.91.116.4:4940     , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.012693621 (518h5m50.883776676s) [noise=true , address=45.91.116.6:4973     , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.012717165 (518h5m50.88380021s) [seg_pos={0 0}, upload=454544]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:34:51.012957062 (518h5m50.884040107s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.012968935 (518h5m50.884051981s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013095285 (518h5m50.884178331s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.013113350 (518h5m50.884196395s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013296098 (518h5m50.884379143s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.013303251 (518h5m50.884386287s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013325995 (518h5m50.88440903s) [seg_pos={0 0}, upload=454544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:34:51.013423311 (518h5m50.884506346s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:34:51.013492492 (518h5m50.884575537s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:34:51.301485099 (518h5m51.172568134s) [upload=454544]: Done uploading segments...
Dec 27 05:34:51.301504687 (518h5m51.172587732s) [upload=454544]: Upload finished. err: <nil>
Duration: 5.811597655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
29 out of 60 is failed (threshold=3) [X.XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX]
duration: 
30.003463762s
trace: 
https://jaeger.dev.storj.io/trace/4170035c95ee3ad0
FAIL (2025-12-27 05:35:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 05:34:48.380482691 (518h5m48.251565726s) [upload=454544]: Starting upload
Dec 27 05:34:48.380562924 (518h5m48.25164596s) [upload=454544]: Done starting upload
Dec 27 05:34:48.380673735 (518h5m48.25175678s) [upload=454544]: Uploading segments...
Dec 27 05:34:48.380782292 (518h5m48.251865337s) [upload=454544]: Got next segment. Inline: false
Dec 27 05:34:48.380815305 (518h5m48.25189835s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 05:34:48.380820384 (518h5m48.251903419s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 05:34:48.626473509 (518h5m48.497556555s) [seg_pos={0 0}, upload=454544]: Begin upload segment...
Dec 27 05:34:48.626528334 (518h5m48.497611369s) [seg_pos={0 0}, upload=454544]: Done begin upload segment.
Dec 27 05:34:48.626556177 (518h5m48.497639212s) [noise=true , address=145.79.227.40:4050   , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.628331186 (518h5m48.499414232s) [noise=true , address=145.79.227.5:4221    , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.628728574 (518h5m48.499811619s) [noise=true , address=145.79.227.49:4338   , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629099119 (518h5m48.500182154s) [noise=true , address=145.79.227.41:4086   , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629419209 (518h5m48.500502244s) [noise=true , address=145.79.227.9:4366    , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.629760218 (518h5m48.500843253s) [noise=true , address=45.91.116.6:4973     , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.630136835 (518h5m48.50121988s) [noise=true , address=145.79.227.37:4566   , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.631218024 (518h5m48.502301069s) [noise=true , address=145.79.227.42:4122   , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.632043446 (518h5m48.503126491s) [noise=true , address=145.79.227.48:4303   , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.632670550 (518h5m48.503753585s) [noise=true , address=45.91.116.4:4940     , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.634500525 (518h5m48.50558357s) [noise=true , address=145.79.227.2:4114    , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.635139160 (518h5m48.506222206s) [noise=true , address=45.91.116.13:4154    , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.635823263 (518h5m48.506906309s) [noise=true , address=145.79.227.36:4518   , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.636522755 (518h5m48.50760579s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.637098070 (518h5m48.508181105s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:48.637850353 (518h5m48.508933388s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Uploading piece...
Dec 27 05:34:50.260369707 (518h5m50.131452752s) [upload=454544]: Next returned nil segment
Dec 27 05:34:50.260382743 (518h5m50.131465778s) [upload=454544]: Waiting for error group managing segments...
Dec 27 05:34:50.639212409 (518h5m50.510295444s) [noise=true , address=145.79.227.5:4221    , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.640973903 (518h5m50.512056938s) [noise=true , address=145.79.227.36:4518   , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.641271821 (518h5m50.512354866s) [noise=true , address=145.79.227.9:4366    , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.641696529 (518h5m50.512779564s) [noise=true , address=145.79.227.41:4086   , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.642399538 (518h5m50.513482583s) [noise=true , address=145.79.227.2:4114    , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.644172935 (518h5m50.51525597s) [noise=true , address=145.79.227.49:4338   , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.651155731 (518h5m50.522238776s) [noise=true , address=145.79.227.40:4050   , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.656354861 (518h5m50.527437896s) [noise=true , address=145.79.227.48:4303   , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.659760366 (518h5m50.530843411s) [noise=true , address=145.79.227.37:4566   , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.672522708 (518h5m50.543605743s) [noise=true , address=145.79.227.42:4122   , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:50.908330215 (518h5m50.77941326s) [noise=true , address=45.91.116.13:4154    , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.000300392 (518h5m50.871383427s) [noise=true , address=45.91.116.4:4940     , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.012693621 (518h5m50.883776676s) [noise=true , address=45.91.116.6:4973     , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil>
Dec 27 05:34:51.012717165 (518h5m50.88380021s) [seg_pos={0 0}, upload=454544]: Segment reached optimal threshold of 13 pieces.
Dec 27 05:34:51.012957062 (518h5m50.884040107s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.012968935 (518h5m50.884051981s) [noise=true , address=45.91.116.5:4298     , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013095285 (518h5m50.884178331s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.013113350 (518h5m50.884196395s) [noise=true , address=45.91.116.9:4264     , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013296098 (518h5m50.884379143s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 27 05:34:51.013303251 (518h5m50.884386287s) [noise=true , address=77.90.31.10:4517     , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold
Dec 27 05:34:51.013325995 (518h5m50.88440903s) [seg_pos={0 0}, upload=454544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 05:34:51.013423311 (518h5m50.884506346s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 05:34:51.013492492 (518h5m50.884575537s) [upload=454544]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 05:34:51.301485099 (518h5m51.172568134s) [upload=454544]: Done uploading segments...
Dec 27 05:34:51.301504687 (518h5m51.172587732s) [upload=454544]: Upload finished. err: <nil>
Duration: 30.003487618s core uplink us-select-1