Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.433637645s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:21:01 UTC) orbiter warning: context deadline exceeded Duration: 3.433665808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X......XX.X.X......X.....X..X.XX.................X..........] duration: 5.849743376s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:20:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 5.849765848s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.651071493s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:20:36 UTC) orbiter warning: context deadline exceeded Duration: 3.651100989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 40.15264852s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:20:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 40.152674109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX......XX.X.X......X.....X..X.XX.................X.........] duration: 6.192100098s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:20:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 6.192142639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX......XX.X.X......X.....X..X.XX.................X........] duration: 6.002570599s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:19:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 6.002600978s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.490699545s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:19:23 UTC) orbiter warning: context deadline exceeded Duration: 3.490755651s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 20.389749332s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:19:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 20.389768599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 33.768991779s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:18:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 33.769044841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..XX......XX.X.X......X.....X..X.XX.................X.......] duration: 5.711416985s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:18:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 5.711445859s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.725160033s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:18:33 UTC) orbiter warning: context deadline exceeded Duration: 3.725203796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...XX......XX.X.X......X.....X..X.XX.................X......] duration: 15.11379607s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:18:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 15.113827651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 35.048382376s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:17:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 35.048409236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.455332419s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:17:49 UTC) orbiter warning: context deadline exceeded Duration: 3.455385239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 36.884969212s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:17:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 36.884995051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X...XX......XX.X.X......X.....X..X.XX.................X.....] duration: 5.818207116s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:17:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 5.818238055s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 5.000885514s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:16:53 UTC) orbiter warning: context deadline exceeded Duration: 5.000913737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.629300106s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:15:59 UTC) orbiter warning: context deadline exceeded Duration: 3.629325844s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX...XX......XX.X.X......X.....X..X.XX.................X....] duration: 5.978474303s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:15:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 5.978505573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 29.496569545s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:15:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 29.496598871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 4.369273311s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:15:18 UTC) orbiter warning: context deadline exceeded Duration: 4.369304821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XX...XX......XX.X.X......X.....X..X.XX.................X...] duration: 6.25673693s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:14:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 6.256765113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 35.700900001s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:14:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 35.700919018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 38.905176836s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:14:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 38.905212975s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 4.103277502s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:14:10 UTC) orbiter warning: context deadline exceeded Duration: 4.103300666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.XX...XX......XX.X.X......X.....X..X.XX.................X..] duration: 5.934782526s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:13:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 5.934813275s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.370377957s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:13:21 UTC) orbiter warning: context deadline exceeded Duration: 3.370421069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.XX...XX......XX.X.X......X.....X..X.XX.................X.] duration: 5.770947231s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:13:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 5.770975896s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X.XX...XX......XX.X.X......X.....X..X.XX.................X] duration: 30.004194952s trace: https://jaeger.dev.storj.io/trace/3375b24df5428439 FAIL (2025-12-27 06:13:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA) context deadline exceeded ======LOGS====== Dec 27 06:12:43.192116646 (518h43m43.063199691s) [upload=455106]: Starting upload Dec 27 06:12:43.192192630 (518h43m43.063275675s) [upload=455106]: Done starting upload Dec 27 06:12:43.192247235 (518h43m43.06333027s) [upload=455106]: Uploading segments... Dec 27 06:12:43.192432788 (518h43m43.063515823s) [upload=455106]: Got next segment. Inline: false Dec 27 06:12:43.192484486 (518h43m43.063567522s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:12:43.192490608 (518h43m43.063573653s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:12:43.438583837 (518h43m43.309666893s) [seg_pos={0 0}, upload=455106]: Begin upload segment... Dec 27 06:12:43.438675222 (518h43m43.309758267s) [seg_pos={0 0}, upload=455106]: Done begin upload segment. Dec 27 06:12:43.438713094 (518h43m43.309796129s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.438739675 (518h43m43.30982271s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439471148 (518h43m43.310554213s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.439999243 (518h43m43.311082288s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.440486581 (518h43m43.311569626s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.441094508 (518h43m43.312177553s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.442535382 (518h43m43.313618417s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.443987888 (518h43m43.315070924s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.445494237 (518h43m43.316577282s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.446970238 (518h43m43.318053273s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.448618736 (518h43m43.319701771s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.450773710 (518h43m43.321856745s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.457064940 (518h43m43.328147975s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.459981533 (518h43m43.331064578s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.462600821 (518h43m43.333683866s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:43.464043628 (518h43m43.335126673s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Uploading piece... Dec 27 06:12:45.156016065 (518h43m45.0270991s) [upload=455106]: Next returned nil segment Dec 27 06:12:45.156039009 (518h43m45.027122044s) [upload=455106]: Waiting for error group managing segments... Dec 27 06:12:45.432208079 (518h43m45.303291114s) [noise=true , address=145.79.227.8:4330 , piece_id=MH5CGP2RQSAB2KDH2IWCZR4JI4N4H6WRQGPHAUKINKTX3XDOFMEQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.443907316 (518h43m45.314990361s) [noise=true , address=145.79.227.9:4366 , piece_id=BGO5SV36D3CBID44DBEK7UZRBQXCOVOPTRZLP3VX6CGUM4TZTXCQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.444337715 (518h43m45.31542075s) [noise=true , address=145.79.227.4:4185 , piece_id=H3SIKHXS7XY3VZLHELC4RE2EVKSOHU54DPKUTETX4WR5WLMM3DVA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.449485407 (518h43m45.320568453s) [noise=true , address=145.79.227.12:4765 , piece_id=54RPN7Z5K66U76AYQBUY6HR3DHF3QHA7ICCMJXHHJ7U4RHNJ5N2A, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.463848247 (518h43m45.334931292s) [noise=true , address=145.79.227.25:4617 , piece_id=NDVIJAKVJX3TJM5HLALMZUYSFZSHZO2GEYXVHHGKGFK63PF22NSQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.467310921 (518h43m45.338393956s) [noise=true , address=145.79.227.49:4337 , piece_id=U7HDS2HSAIVMNTKZCN72WKPM5HFHL3X5ROPUSUT2Q5QLJPMBTEZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.468756764 (518h43m45.339839799s) [noise=true , address=145.79.227.28:4727 , piece_id=SA52YRHSZIUXL3SMKQJPLU2IMMOL7ZABJYZQZ2Q37GQO3M6WTSMA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.469893529 (518h43m45.340976564s) [noise=true , address=145.79.227.41:4085 , piece_id=GTIWJZVRRFTIFSUOUPZ22IS7TS7ZBGX2TZLHGWNXD4F4WNRHPEGA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.472762392 (518h43m45.343845427s) [noise=true , address=145.79.227.45:4230 , piece_id=ZBEE3ZKNGP37ZJZSD4YGLGLRR3VJGNRXO5NO24AOBZ5LLNSL4SMQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.486883732 (518h43m45.357966767s) [noise=true , address=145.79.227.51:4410 , piece_id=4M55BY7Y5AXH275WPMIAGVSMELYS7GNLNN2RO5LDXEHBE4QN2PKA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.668909885 (518h43m45.53999292s) [noise=true , address=77.90.31.15:4697 , piece_id=HNXSFCPVOAJXQFBTGPSIIXYGFWUEDQGBREHN34UT3P7TD73SPHZQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:45.767929568 (518h43m45.639012603s) [noise=true , address=77.90.31.14:4663 , piece_id=V6AF3DSGECXGUNKI4BIMBRDYVCZ66DCD4RDHGJQO7K56LHEI3SRA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.409973735 (518h43m46.28105677s) [noise=true , address=45.91.116.7:4012 , piece_id=HJXRTYVMMQB5TXQPC5USFYL3PAUUCYAIRD5DXO2B6DD7QJAX54EA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: <nil> Dec 27 06:12:46.410001218 (518h43m46.281084263s) [seg_pos={0 0}, upload=455106]: Segment reached optimal threshold of 13 pieces. Dec 27 06:12:46.410230575 (518h43m46.28131362s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410238660 (518h43m46.281321705s) [noise=true , address=77.90.31.12:4589 , piece_id=DZXJZHT5ZH5MDDBSPRDTMMNAKGURCH4OXLNN55PNQDZ6RPTTAQTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410494998 (518h43m46.281578043s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410520607 (518h43m46.281603662s) [noise=true , address=45.91.116.14:4190 , piece_id=OQ6A5Y3YKP32DLCEGT7V3RLQR2EJT3WFOM4KASHKMW754RHJJKCA, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410541176 (518h43m46.281624332s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:12:46.410554652 (518h43m46.281637697s) [noise=true , address=77.90.31.11:4555 , piece_id=WQKYHOMKP34UZWVYU6IAZQACXKXVKBRTHGUAAOSHGGSQXNBWJSTQ, seg_pos={0 0}, upload=455106]: Long tail cancellation due to optimal threshold Dec 27 06:12:46.410660874 (518h43m46.281743909s) [seg_pos={0 0}, upload=455106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:12:46.410833823 (518h43m46.281916858s) [upload=455106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:12:46.410841297 (518h43m46.281924342s) [upload=455106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:12:46.718785253 (518h43m46.589868298s) [upload=455106]: Done uploading segments... Dec 27 06:12:46.718810040 (518h43m46.589893075s) [upload=455106]: Upload finished. err: <nil> Duration: 30.004223977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 21.564493865s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:12:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 21.564518611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 39.013472637s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:12:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 39.013491723s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.551952413s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:12:08 UTC) orbiter warning: context deadline exceeded Duration: 3.551983622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.X.XX...XX......XX.X.X......X.....X..X.XX.................] duration: 5.900617208s trace: https://jaeger.dev.storj.io/trace/6e7b1f1cd9efe572 FAIL (2025-12-27 06:11:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA) context deadline exceeded ======LOGS====== 05:58:54.027618010 (518h29m53.898701045s) [upload=454908]: Starting upload Dec 27 05:58:54.027674568 (518h29m53.898757613s) [upload=454908]: Done starting upload Dec 27 05:58:54.027752656 (518h29m53.898835691s) [upload=454908]: Uploading segments... Dec 27 05:58:54.027867967 (518h29m53.898951002s) [upload=454908]: Got next segment. Inline: false Dec 27 05:58:54.027959360 (518h29m53.899042395s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:58:54.027966052 (518h29m53.899049088s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:58:54.273197646 (518h29m54.144280681s) [seg_pos={0 0}, upload=454908]: Begin upload segment... Dec 27 05:58:54.273253432 (518h29m54.144336467s) [seg_pos={0 0}, upload=454908]: Done begin upload segment. Dec 27 05:58:54.273335729 (518h29m54.144418764s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.273355857 (518h29m54.144438902s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.274397771 (518h29m54.145480816s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275285933 (518h29m54.146368968s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275978290 (518h29m54.147061335s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.277650785 (518h29m54.14873382s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.281356081 (518h29m54.152439116s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.283248223 (518h29m54.154331268s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.286543088 (518h29m54.157626123s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.296837890 (518h29m54.167920935s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.297626512 (518h29m54.168709547s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.298856263 (518h29m54.169939298s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.299710430 (518h29m54.170793485s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.300773284 (518h29m54.171856319s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.301333490 (518h29m54.172416535s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.302620932 (518h29m54.173703967s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:55.915655523 (518h29m55.786738558s) [upload=454908]: Next returned nil segment Dec 27 05:58:55.915665622 (518h29m55.786748657s) [upload=454908]: Waiting for error group managing segments... Dec 27 05:58:56.302185873 (518h29m56.173268918s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305487398 (518h29m56.176570434s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305946472 (518h29m56.177029507s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.310843968 (518h29m56.181927003s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.313521086 (518h29m56.184604141s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317048212 (518h29m56.188131247s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317491997 (518h29m56.188575043s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.327595456 (518h29m56.198678491s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.332749460 (518h29m56.203832505s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.467173567 (518h29m56.338256602s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.555793674 (518h29m56.426876719s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.612756451 (518h29m56.483839486s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663674767 (518h29m56.534757813s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663704895 (518h29m56.53478794s) [seg_pos={0 0}, upload=454908]: Segment reached optimal threshold of 13 pieces. Dec 27 05:58:56.663995689 (518h29m56.535078754s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection Dec 27 05:58:56.664001861 (518h29m56.535084906s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664126127 (518h29m56.535209172s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664169029 (518h29m56.535252065s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664391382 (518h29m56.535474418s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664398546 (518h29m56.535481591s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664419476 (518h29m56.535502521s) [seg_pos={0 0}, upload=454908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:58:56.664503536 (518h29m56.535586571s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:58:56.664509056 (518h29m56.535592091s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:58:56.958441263 (518h29m56.829524308s) [upload=454908]: Done uploading segments... Dec 27 05:58:56.958480828 (518h29m56.829563863s) [upload=454908]: Upload finished. err: <nil> Duration: 5.900642647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 46.4873369s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:11:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 46.487369543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX.X.XX...XX......XX.X.X......X.....X..X.XX................] duration: 29.886275898s trace: https://jaeger.dev.storj.io/trace/6e7b1f1cd9efe572 FAIL (2025-12-27 06:11:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA) context deadline exceeded ======LOGS====== 05:58:54.027618010 (518h29m53.898701045s) [upload=454908]: Starting upload Dec 27 05:58:54.027674568 (518h29m53.898757613s) [upload=454908]: Done starting upload Dec 27 05:58:54.027752656 (518h29m53.898835691s) [upload=454908]: Uploading segments... Dec 27 05:58:54.027867967 (518h29m53.898951002s) [upload=454908]: Got next segment. Inline: false Dec 27 05:58:54.027959360 (518h29m53.899042395s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:58:54.027966052 (518h29m53.899049088s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:58:54.273197646 (518h29m54.144280681s) [seg_pos={0 0}, upload=454908]: Begin upload segment... Dec 27 05:58:54.273253432 (518h29m54.144336467s) [seg_pos={0 0}, upload=454908]: Done begin upload segment. Dec 27 05:58:54.273335729 (518h29m54.144418764s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.273355857 (518h29m54.144438902s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.274397771 (518h29m54.145480816s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275285933 (518h29m54.146368968s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275978290 (518h29m54.147061335s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.277650785 (518h29m54.14873382s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.281356081 (518h29m54.152439116s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.283248223 (518h29m54.154331268s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.286543088 (518h29m54.157626123s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.296837890 (518h29m54.167920935s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.297626512 (518h29m54.168709547s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.298856263 (518h29m54.169939298s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.299710430 (518h29m54.170793485s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.300773284 (518h29m54.171856319s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.301333490 (518h29m54.172416535s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.302620932 (518h29m54.173703967s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:55.915655523 (518h29m55.786738558s) [upload=454908]: Next returned nil segment Dec 27 05:58:55.915665622 (518h29m55.786748657s) [upload=454908]: Waiting for error group managing segments... Dec 27 05:58:56.302185873 (518h29m56.173268918s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305487398 (518h29m56.176570434s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305946472 (518h29m56.177029507s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.310843968 (518h29m56.181927003s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.313521086 (518h29m56.184604141s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317048212 (518h29m56.188131247s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317491997 (518h29m56.188575043s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.327595456 (518h29m56.198678491s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.332749460 (518h29m56.203832505s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.467173567 (518h29m56.338256602s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.555793674 (518h29m56.426876719s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.612756451 (518h29m56.483839486s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663674767 (518h29m56.534757813s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663704895 (518h29m56.53478794s) [seg_pos={0 0}, upload=454908]: Segment reached optimal threshold of 13 pieces. Dec 27 05:58:56.663995689 (518h29m56.535078754s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection Dec 27 05:58:56.664001861 (518h29m56.535084906s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664126127 (518h29m56.535209172s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664169029 (518h29m56.535252065s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664391382 (518h29m56.535474418s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664398546 (518h29m56.535481591s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664419476 (518h29m56.535502521s) [seg_pos={0 0}, upload=454908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:58:56.664503536 (518h29m56.535586571s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:58:56.664509056 (518h29m56.535592091s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:58:56.958441263 (518h29m56.829524308s) [upload=454908]: Done uploading segments... Dec 27 05:58:56.958480828 (518h29m56.829563863s) [upload=454908]: Upload finished. err: <nil> Duration: 29.886308769s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.585232104s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:11:40 UTC) orbiter warning: context deadline exceeded Duration: 3.58526058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XX.X.XX...XX......XX.X.X......X.....X..X.XX...............] duration: 5.76643157s trace: https://jaeger.dev.storj.io/trace/6e7b1f1cd9efe572 FAIL (2025-12-27 06:10:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA) context deadline exceeded ======LOGS====== 05:58:54.027618010 (518h29m53.898701045s) [upload=454908]: Starting upload Dec 27 05:58:54.027674568 (518h29m53.898757613s) [upload=454908]: Done starting upload Dec 27 05:58:54.027752656 (518h29m53.898835691s) [upload=454908]: Uploading segments... Dec 27 05:58:54.027867967 (518h29m53.898951002s) [upload=454908]: Got next segment. Inline: false Dec 27 05:58:54.027959360 (518h29m53.899042395s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:58:54.027966052 (518h29m53.899049088s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:58:54.273197646 (518h29m54.144280681s) [seg_pos={0 0}, upload=454908]: Begin upload segment... Dec 27 05:58:54.273253432 (518h29m54.144336467s) [seg_pos={0 0}, upload=454908]: Done begin upload segment. Dec 27 05:58:54.273335729 (518h29m54.144418764s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.273355857 (518h29m54.144438902s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.274397771 (518h29m54.145480816s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275285933 (518h29m54.146368968s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275978290 (518h29m54.147061335s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.277650785 (518h29m54.14873382s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.281356081 (518h29m54.152439116s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.283248223 (518h29m54.154331268s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.286543088 (518h29m54.157626123s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.296837890 (518h29m54.167920935s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.297626512 (518h29m54.168709547s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.298856263 (518h29m54.169939298s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.299710430 (518h29m54.170793485s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.300773284 (518h29m54.171856319s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.301333490 (518h29m54.172416535s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.302620932 (518h29m54.173703967s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:55.915655523 (518h29m55.786738558s) [upload=454908]: Next returned nil segment Dec 27 05:58:55.915665622 (518h29m55.786748657s) [upload=454908]: Waiting for error group managing segments... Dec 27 05:58:56.302185873 (518h29m56.173268918s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305487398 (518h29m56.176570434s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305946472 (518h29m56.177029507s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.310843968 (518h29m56.181927003s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.313521086 (518h29m56.184604141s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317048212 (518h29m56.188131247s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317491997 (518h29m56.188575043s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.327595456 (518h29m56.198678491s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.332749460 (518h29m56.203832505s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.467173567 (518h29m56.338256602s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.555793674 (518h29m56.426876719s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.612756451 (518h29m56.483839486s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663674767 (518h29m56.534757813s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663704895 (518h29m56.53478794s) [seg_pos={0 0}, upload=454908]: Segment reached optimal threshold of 13 pieces. Dec 27 05:58:56.663995689 (518h29m56.535078754s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection Dec 27 05:58:56.664001861 (518h29m56.535084906s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664126127 (518h29m56.535209172s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664169029 (518h29m56.535252065s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664391382 (518h29m56.535474418s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664398546 (518h29m56.535481591s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664419476 (518h29m56.535502521s) [seg_pos={0 0}, upload=454908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:58:56.664503536 (518h29m56.535586571s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:58:56.664509056 (518h29m56.535592091s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:58:56.958441263 (518h29m56.829524308s) [upload=454908]: Done uploading segments... Dec 27 05:58:56.958480828 (518h29m56.829563863s) [upload=454908]: Upload finished. err: <nil> Duration: 5.766457469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 24.656958414s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:10:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 24.65698748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.550426426s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:10:31 UTC) orbiter warning: context deadline exceeded Duration: 3.550493525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..XX.X.XX...XX......XX.X.X......X.....X..X.XX..............] duration: 5.761897368s trace: https://jaeger.dev.storj.io/trace/6e7b1f1cd9efe572 FAIL (2025-12-27 06:09:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA) context deadline exceeded ======LOGS====== 05:58:54.027618010 (518h29m53.898701045s) [upload=454908]: Starting upload Dec 27 05:58:54.027674568 (518h29m53.898757613s) [upload=454908]: Done starting upload Dec 27 05:58:54.027752656 (518h29m53.898835691s) [upload=454908]: Uploading segments... Dec 27 05:58:54.027867967 (518h29m53.898951002s) [upload=454908]: Got next segment. Inline: false Dec 27 05:58:54.027959360 (518h29m53.899042395s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:58:54.027966052 (518h29m53.899049088s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:58:54.273197646 (518h29m54.144280681s) [seg_pos={0 0}, upload=454908]: Begin upload segment... Dec 27 05:58:54.273253432 (518h29m54.144336467s) [seg_pos={0 0}, upload=454908]: Done begin upload segment. Dec 27 05:58:54.273335729 (518h29m54.144418764s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.273355857 (518h29m54.144438902s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.274397771 (518h29m54.145480816s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275285933 (518h29m54.146368968s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275978290 (518h29m54.147061335s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.277650785 (518h29m54.14873382s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.281356081 (518h29m54.152439116s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.283248223 (518h29m54.154331268s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.286543088 (518h29m54.157626123s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.296837890 (518h29m54.167920935s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.297626512 (518h29m54.168709547s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.298856263 (518h29m54.169939298s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.299710430 (518h29m54.170793485s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.300773284 (518h29m54.171856319s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.301333490 (518h29m54.172416535s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.302620932 (518h29m54.173703967s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:55.915655523 (518h29m55.786738558s) [upload=454908]: Next returned nil segment Dec 27 05:58:55.915665622 (518h29m55.786748657s) [upload=454908]: Waiting for error group managing segments... Dec 27 05:58:56.302185873 (518h29m56.173268918s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305487398 (518h29m56.176570434s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305946472 (518h29m56.177029507s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.310843968 (518h29m56.181927003s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.313521086 (518h29m56.184604141s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317048212 (518h29m56.188131247s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317491997 (518h29m56.188575043s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.327595456 (518h29m56.198678491s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.332749460 (518h29m56.203832505s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.467173567 (518h29m56.338256602s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.555793674 (518h29m56.426876719s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.612756451 (518h29m56.483839486s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663674767 (518h29m56.534757813s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663704895 (518h29m56.53478794s) [seg_pos={0 0}, upload=454908]: Segment reached optimal threshold of 13 pieces. Dec 27 05:58:56.663995689 (518h29m56.535078754s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection Dec 27 05:58:56.664001861 (518h29m56.535084906s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664126127 (518h29m56.535209172s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664169029 (518h29m56.535252065s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664391382 (518h29m56.535474418s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664398546 (518h29m56.535481591s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664419476 (518h29m56.535502521s) [seg_pos={0 0}, upload=454908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:58:56.664503536 (518h29m56.535586571s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:58:56.664509056 (518h29m56.535592091s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:58:56.958441263 (518h29m56.829524308s) [upload=454908]: Done uploading segments... Dec 27 05:58:56.958480828 (518h29m56.829563863s) [upload=454908]: Upload finished. err: <nil> Duration: 5.761915723s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.41523625s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:09:50 UTC) orbiter warning: context deadline exceeded Duration: 3.415255698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.............X.............................................] duration: 28.730688896s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:09:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 28.730732589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X..XX.X.XX...XX......XX.X.X......X.....X..X.XX.............] duration: 5.864551752s trace: https://jaeger.dev.storj.io/trace/6e7b1f1cd9efe572 FAIL (2025-12-27 06:09:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA) context deadline exceeded ======LOGS====== 05:58:54.027618010 (518h29m53.898701045s) [upload=454908]: Starting upload Dec 27 05:58:54.027674568 (518h29m53.898757613s) [upload=454908]: Done starting upload Dec 27 05:58:54.027752656 (518h29m53.898835691s) [upload=454908]: Uploading segments... Dec 27 05:58:54.027867967 (518h29m53.898951002s) [upload=454908]: Got next segment. Inline: false Dec 27 05:58:54.027959360 (518h29m53.899042395s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:58:54.027966052 (518h29m53.899049088s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:58:54.273197646 (518h29m54.144280681s) [seg_pos={0 0}, upload=454908]: Begin upload segment... Dec 27 05:58:54.273253432 (518h29m54.144336467s) [seg_pos={0 0}, upload=454908]: Done begin upload segment. Dec 27 05:58:54.273335729 (518h29m54.144418764s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.273355857 (518h29m54.144438902s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.274397771 (518h29m54.145480816s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275285933 (518h29m54.146368968s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275978290 (518h29m54.147061335s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.277650785 (518h29m54.14873382s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.281356081 (518h29m54.152439116s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.283248223 (518h29m54.154331268s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.286543088 (518h29m54.157626123s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.296837890 (518h29m54.167920935s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.297626512 (518h29m54.168709547s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.298856263 (518h29m54.169939298s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.299710430 (518h29m54.170793485s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.300773284 (518h29m54.171856319s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.301333490 (518h29m54.172416535s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.302620932 (518h29m54.173703967s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:55.915655523 (518h29m55.786738558s) [upload=454908]: Next returned nil segment Dec 27 05:58:55.915665622 (518h29m55.786748657s) [upload=454908]: Waiting for error group managing segments... Dec 27 05:58:56.302185873 (518h29m56.173268918s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305487398 (518h29m56.176570434s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305946472 (518h29m56.177029507s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.310843968 (518h29m56.181927003s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.313521086 (518h29m56.184604141s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317048212 (518h29m56.188131247s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317491997 (518h29m56.188575043s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.327595456 (518h29m56.198678491s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.332749460 (518h29m56.203832505s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.467173567 (518h29m56.338256602s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.555793674 (518h29m56.426876719s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.612756451 (518h29m56.483839486s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663674767 (518h29m56.534757813s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663704895 (518h29m56.53478794s) [seg_pos={0 0}, upload=454908]: Segment reached optimal threshold of 13 pieces. Dec 27 05:58:56.663995689 (518h29m56.535078754s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection Dec 27 05:58:56.664001861 (518h29m56.535084906s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664126127 (518h29m56.535209172s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664169029 (518h29m56.535252065s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664391382 (518h29m56.535474418s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664398546 (518h29m56.535481591s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664419476 (518h29m56.535502521s) [seg_pos={0 0}, upload=454908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:58:56.664503536 (518h29m56.535586571s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:58:56.664509056 (518h29m56.535592091s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:58:56.958441263 (518h29m56.829524308s) [upload=454908]: Done uploading segments... Dec 27 05:58:56.958480828 (518h29m56.829563863s) [upload=454908]: Upload finished. err: <nil> Duration: 5.864581458s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.531125313s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:09:01 UTC) orbiter warning: context deadline exceeded Duration: 3.531164086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.............X............................................] duration: 36.448703291s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:08:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 36.44872858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X..XX.X.XX...XX......XX.X.X......X.....X..X.XX............] duration: 5.785291523s trace: https://jaeger.dev.storj.io/trace/6e7b1f1cd9efe572 FAIL (2025-12-27 06:08:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA) context deadline exceeded ======LOGS====== 05:58:54.027618010 (518h29m53.898701045s) [upload=454908]: Starting upload Dec 27 05:58:54.027674568 (518h29m53.898757613s) [upload=454908]: Done starting upload Dec 27 05:58:54.027752656 (518h29m53.898835691s) [upload=454908]: Uploading segments... Dec 27 05:58:54.027867967 (518h29m53.898951002s) [upload=454908]: Got next segment. Inline: false Dec 27 05:58:54.027959360 (518h29m53.899042395s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:58:54.027966052 (518h29m53.899049088s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:58:54.273197646 (518h29m54.144280681s) [seg_pos={0 0}, upload=454908]: Begin upload segment... Dec 27 05:58:54.273253432 (518h29m54.144336467s) [seg_pos={0 0}, upload=454908]: Done begin upload segment. Dec 27 05:58:54.273335729 (518h29m54.144418764s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.273355857 (518h29m54.144438902s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.274397771 (518h29m54.145480816s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275285933 (518h29m54.146368968s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275978290 (518h29m54.147061335s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.277650785 (518h29m54.14873382s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.281356081 (518h29m54.152439116s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.283248223 (518h29m54.154331268s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.286543088 (518h29m54.157626123s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.296837890 (518h29m54.167920935s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.297626512 (518h29m54.168709547s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.298856263 (518h29m54.169939298s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.299710430 (518h29m54.170793485s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.300773284 (518h29m54.171856319s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.301333490 (518h29m54.172416535s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.302620932 (518h29m54.173703967s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:55.915655523 (518h29m55.786738558s) [upload=454908]: Next returned nil segment Dec 27 05:58:55.915665622 (518h29m55.786748657s) [upload=454908]: Waiting for error group managing segments... Dec 27 05:58:56.302185873 (518h29m56.173268918s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305487398 (518h29m56.176570434s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305946472 (518h29m56.177029507s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.310843968 (518h29m56.181927003s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.313521086 (518h29m56.184604141s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317048212 (518h29m56.188131247s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317491997 (518h29m56.188575043s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.327595456 (518h29m56.198678491s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.332749460 (518h29m56.203832505s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.467173567 (518h29m56.338256602s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.555793674 (518h29m56.426876719s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.612756451 (518h29m56.483839486s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663674767 (518h29m56.534757813s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663704895 (518h29m56.53478794s) [seg_pos={0 0}, upload=454908]: Segment reached optimal threshold of 13 pieces. Dec 27 05:58:56.663995689 (518h29m56.535078754s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection Dec 27 05:58:56.664001861 (518h29m56.535084906s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664126127 (518h29m56.535209172s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664169029 (518h29m56.535252065s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664391382 (518h29m56.535474418s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664398546 (518h29m56.535481591s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664419476 (518h29m56.535502521s) [seg_pos={0 0}, upload=454908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:58:56.664503536 (518h29m56.535586571s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:58:56.664509056 (518h29m56.535592091s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:58:56.958441263 (518h29m56.829524308s) [upload=454908]: Done uploading segments... Dec 27 05:58:56.958480828 (518h29m56.829563863s) [upload=454908]: Upload finished. err: <nil> Duration: 5.785328915s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.602254762s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:07:47 UTC) orbiter warning: context deadline exceeded Duration: 3.602282715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.............X...........................................] duration: 15.353427118s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 06:07:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 15.353456274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.449272338s trace: https://jaeger.dev.storj.io/trace/39d952b81959c79e FAIL (2025-12-27 06:07:22 UTC) orbiter warning: context deadline exceeded Duration: 3.44935764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X..XX.X.XX...XX......XX.X.X......X.....X..X.XX...........] duration: 6.396986467s trace: https://jaeger.dev.storj.io/trace/6e7b1f1cd9efe572 FAIL (2025-12-27 06:07:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA) context deadline exceeded ======LOGS====== 05:58:54.027618010 (518h29m53.898701045s) [upload=454908]: Starting upload Dec 27 05:58:54.027674568 (518h29m53.898757613s) [upload=454908]: Done starting upload Dec 27 05:58:54.027752656 (518h29m53.898835691s) [upload=454908]: Uploading segments... Dec 27 05:58:54.027867967 (518h29m53.898951002s) [upload=454908]: Got next segment. Inline: false Dec 27 05:58:54.027959360 (518h29m53.899042395s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:58:54.027966052 (518h29m53.899049088s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:58:54.273197646 (518h29m54.144280681s) [seg_pos={0 0}, upload=454908]: Begin upload segment... Dec 27 05:58:54.273253432 (518h29m54.144336467s) [seg_pos={0 0}, upload=454908]: Done begin upload segment. Dec 27 05:58:54.273335729 (518h29m54.144418764s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.273355857 (518h29m54.144438902s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.274397771 (518h29m54.145480816s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275285933 (518h29m54.146368968s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.275978290 (518h29m54.147061335s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.277650785 (518h29m54.14873382s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.281356081 (518h29m54.152439116s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.283248223 (518h29m54.154331268s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.286543088 (518h29m54.157626123s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.296837890 (518h29m54.167920935s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.297626512 (518h29m54.168709547s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.298856263 (518h29m54.169939298s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.299710430 (518h29m54.170793485s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.300773284 (518h29m54.171856319s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.301333490 (518h29m54.172416535s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:54.302620932 (518h29m54.173703967s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Uploading piece... Dec 27 05:58:55.915655523 (518h29m55.786738558s) [upload=454908]: Next returned nil segment Dec 27 05:58:55.915665622 (518h29m55.786748657s) [upload=454908]: Waiting for error group managing segments... Dec 27 05:58:56.302185873 (518h29m56.173268918s) [noise=true , address=145.79.227.28:4725 , piece_id=3W7B6G66JTVYMXFX6DVDNVVLA6VB3DNZST7HQHCUCGSDVTUAAY5Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305487398 (518h29m56.176570434s) [noise=true , address=145.79.227.45:4231 , piece_id=2O4K3L55UBRWUP3TSQM5JY3NHYS6YK3BJKWLCG3OMRKCDYLH4LJA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.305946472 (518h29m56.177029507s) [noise=true , address=145.79.227.17:4971 , piece_id=SBFPZDVWPSBTJDNPRKQYLXGMGU46NLZKL5ANYBK4YMK72N2FZ7MQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.310843968 (518h29m56.181927003s) [noise=true , address=145.79.227.50:4374 , piece_id=PD7SJTR3S2BRT6VCD5Q6NF5MI4EEO564G4GNXBYTWJREMOOHLOHQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.313521086 (518h29m56.184604141s) [noise=true , address=145.79.227.38:4978 , piece_id=OUSSJZKPVLSUIWFBRNJ2XGOKR6BDNLTN5JRCOPYFPPZMPVUQCNFQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317048212 (518h29m56.188131247s) [noise=true , address=145.79.227.39:4013 , piece_id=VG37JHKAJ5QZK7D72YCMDIYPOQJZOZVLS5BQNFYWE7GSDZUZ4DBA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.317491997 (518h29m56.188575043s) [noise=true , address=145.79.227.31:4797 , piece_id=CEGDRSEKGLLLYK6HWYX6B5P2E3HJ4ZISJFFRAO4VCJ3VDFDXJJ7A, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.327595456 (518h29m56.198678491s) [noise=true , address=145.79.227.6:4259 , piece_id=W6332SDFZD4PH77I7NIQ2UJO5LJ35SYLIRQPS7FOXIEAC563XIMQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.332749460 (518h29m56.203832505s) [noise=true , address=145.79.227.46:4603 , piece_id=FOUSW2IBC6KJ7BCMMKLEFB4LQ4QSJZO6V2TNMHKJUQVVTKABK4BQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.467173567 (518h29m56.338256602s) [noise=true , address=77.90.31.14:4661 , piece_id=XMJAOOLKXNSUHXS6WAC6DFEQ5JPQGQLUXJY56GVE436UHGOTVCJQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.555793674 (518h29m56.426876719s) [noise=true , address=45.91.116.7:4009 , piece_id=T52MUHMIDBJC2QMFRQBGR7WOGGL3VPYFUIXBXUUM3BOBYI3PFURA, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.612756451 (518h29m56.483839486s) [noise=true , address=77.90.31.11:4556 , piece_id=RVDYCDC3DNCOT7E4TJ6RQQBBUDZDTYMAMPXGCF6P3OZF6GRLG4RQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663674767 (518h29m56.534757813s) [noise=true , address=45.91.116.10:4084 , piece_id=ZMPVKGAO6BC5NSEJLNCBPKCRRN7UBWQRA5QZKR6VDT5GCIFUKG3Q, seg_pos={0 0}, upload=454908]: Done uploading piece. err: <nil> Dec 27 05:58:56.663704895 (518h29m56.53478794s) [seg_pos={0 0}, upload=454908]: Segment reached optimal threshold of 13 pieces. Dec 27 05:58:56.663995689 (518h29m56.535078754s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->77.90.31.19:4842: use of closed network connection Dec 27 05:58:56.664001861 (518h29m56.535084906s) [noise=true , address=77.90.31.19:4842 , piece_id=4XEBJFGHO7GNC633MV2BGQU24E2TMTYSVXH7BLSGSTE6NJX4ALPQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664126127 (518h29m56.535209172s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664169029 (518h29m56.535252065s) [noise=true , address=45.91.116.12:4226 , piece_id=PDR7XQCJ7TMMJQJDWZHXRP35XYQFA4D4YYSFOTCGUXLJ77KQKBOQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664391382 (518h29m56.535474418s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 05:58:56.664398546 (518h29m56.535481591s) [noise=true , address=77.90.31.21:4877 , piece_id=DLDBWOTXDBIXI7ONED3KVH7UKJLN4J7B73NS6TLOVUMDD7ACXRCQ, seg_pos={0 0}, upload=454908]: Long tail cancellation due to optimal threshold Dec 27 05:58:56.664419476 (518h29m56.535502521s) [seg_pos={0 0}, upload=454908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:58:56.664503536 (518h29m56.535586571s) [upload=454908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:58:56.664509056 (518h29m56.535592091s) [upload=454908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:58:56.958441263 (518h29m56.829524308s) [upload=454908]: Done uploading segments... Dec 27 05:58:56.958480828 (518h29m56.829563863s) [upload=454908]: Upload finished. err: <nil> Duration: 6.397017166s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)