Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 905644 reports (Page 11269 of 18113)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............X......X.....................................]
duration: 
30.946553208s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:25:06 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 30.94656985s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [.X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X.X...XXX.X]
duration: 
30.002922498s
trace: 
https://jaeger.dev.storj.io/trace/4bfe9eeeadd520cd
FAIL (2025-12-30 03:24:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: JICWFBYEC7EGLMT5K6CC25AJEPLK5WLXMO2MOHST435YN56ZMECQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 2EZ6PVGQ3U4AKFFXA7R7IWEHX7LDBGOAOXBQIRMCDBB3FII5W7QA) context deadline exceeded
======LOGS======
ec 30 03:24:22.894839052 (587h55m22.765922107s) [upload=515382]: Got next segment. Inline: false
Dec 30 03:24:22.894867647 (587h55m22.765950692s) [upload=515382]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:24:22.894872546 (587h55m22.765955581s) [upload=515382]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:24:23.146055022 (587h55m23.017138077s) [seg_pos={0 0}, upload=515382]: Begin upload segment...
Dec 30 03:24:23.146210047 (587h55m23.017293092s) [seg_pos={0 0}, upload=515382]: Done begin upload segment.
Dec 30 03:24:23.146271304 (587h55m23.017354349s) [noise=true , address=45.91.116.7:4011     , piece_id=3PNYKM4ZZUET4CUAEGS47NHOCG547ESP6PNAKYUPGB5K6DF3LUNQ, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.146293717 (587h55m23.017376762s) [noise=true , address=145.79.227.16:4909   , piece_id=YMXXFV7DE3JSTF5B4JB2MBTVF2SPK5X5SPIRY33T67MPZ7TZQCNA, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.147014700 (587h55m23.018097735s) [noise=true , address=145.79.227.41:4087   , piece_id=CS4CJR5LSLRUNR35VNYHJYWMO33I2IQDH6KLY5Y5RGR7OAY7OFVQ, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.147053153 (587h55m23.018136188s) [noise=true , address=77.90.31.10:4520     , piece_id=PWGM5QHD7NEXABJK5BLRWMVW26BFKFADAKGEGZ2CVPGQIKWVP4VA, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.147380707 (587h55m23.018463742s) [noise=true , address=145.79.227.34:4906   , piece_id=P7R2FYPDYLWZUPD32JI2KYP3DNQMZOQ5HQGAC2D7DJBXIZ3MHCNA, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.147864438 (587h55m23.018947483s) [noise=true , address=145.79.227.13:4802   , piece_id=O4XNBEXY2K6DWPW3A3LGRTYHJHPVIRCO55CP7ZD2FM7DP3RYHEUA, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.148370782 (587h55m23.019453817s) [noise=true , address=145.79.227.2:4115    , piece_id=XWI2OR5BYOJLDBH6FNONYWY4Q7MA3OMZL32ZYQ2627P7SAOALYBA, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.148885902 (587h55m23.019968948s) [noise=true , address=145.79.227.37:4565   , piece_id=XSSZN3A7BCL77K72MJ3BZ2VPUBLAQYA5BOZX5JA2W3U4B2EWYPEQ, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.149184321 (587h55m23.020267366s) [noise=true , address=77.90.31.16:4735     , piece_id=WR6T6EJTAOENHFF5IGSM6NTI5GBMHANLCCL5XGKXCKV4OBU2GXQA, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.149635841 (587h55m23.020718886s) [noise=true , address=145.79.227.45:4229   , piece_id=DC7P72TE7EEJVPTFRL5RVUIESPDQOAUVO627NQTK5WLG4YSYSLVQ, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.150190157 (587h55m23.021273192s) [noise=true , address=77.90.31.19:4844     , piece_id=KOLINXXA6TQ62BCUVZ2R26C5HLBBDORJKKUSC2B746WBDPXVIAGA, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.165586463 (587h55m23.036669509s) [noise=true , address=145.79.227.46:4603   , piece_id=4AZEFNUK4ZNACF3W6RUKJMOQ2F6LZWXBLW73AR5FVFYKBF5QCYHQ, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.169900347 (587h55m23.040983382s) [noise=true , address=45.91.116.12:4225    , piece_id=2EZ6PVGQ3U4AKFFXA7R7IWEHX7LDBGOAOXBQIRMCDBB3FII5W7QA, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.170859273 (587h55m23.041942318s) [noise=true , address=145.79.227.42:4123   , piece_id=IEHPTP4N7477FC3JV4ZWUHTZTBWEO5VOXOHDO7YIO46BM3EKKCRQ, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.173196944 (587h55m23.04427999s) [noise=true , address=77.90.31.17:4773     , piece_id=JICWFBYEC7EGLMT5K6CC25AJEPLK5WLXMO2MOHST435YN56ZMECQ, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:23.174368024 (587h55m23.045451069s) [noise=true , address=145.79.227.50:4374   , piece_id=TZQTA3K6K3MTL4RE4O4GFOGNHQGAGQ2OSA23GVX2D7I5NBDXC7YA, seg_pos={0 0}, upload=515382]: Uploading piece...
Dec 30 03:24:24.749353703 (587h55m24.620436748s) [upload=515382]: Next returned nil segment
Dec 30 03:24:24.749369444 (587h55m24.620452489s) [upload=515382]: Waiting for error group managing segments...
Dec 30 03:24:25.130541959 (587h55m25.001625004s) [noise=true , address=145.79.227.45:4229   , piece_id=DC7P72TE7EEJVPTFRL5RVUIESPDQOAUVO627NQTK5WLG4YSYSLVQ, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:25.135141447 (587h55m25.006224482s) [noise=true , address=145.79.227.16:4909   , piece_id=YMXXFV7DE3JSTF5B4JB2MBTVF2SPK5X5SPIRY33T67MPZ7TZQCNA, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:25.143512457 (587h55m25.014595502s) [noise=true , address=145.79.227.13:4802   , piece_id=O4XNBEXY2K6DWPW3A3LGRTYHJHPVIRCO55CP7ZD2FM7DP3RYHEUA, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:25.144414454 (587h55m25.015497499s) [noise=true , address=145.79.227.41:4087   , piece_id=CS4CJR5LSLRUNR35VNYHJYWMO33I2IQDH6KLY5Y5RGR7OAY7OFVQ, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:25.152628585 (587h55m25.02371162s) [noise=true , address=145.79.227.2:4115    , piece_id=XWI2OR5BYOJLDBH6FNONYWY4Q7MA3OMZL32ZYQ2627P7SAOALYBA, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:25.173051473 (587h55m25.044134518s) [noise=true , address=145.79.227.46:4603   , piece_id=4AZEFNUK4ZNACF3W6RUKJMOQ2F6LZWXBLW73AR5FVFYKBF5QCYHQ, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:25.186271696 (587h55m25.057354741s) [noise=true , address=145.79.227.42:4123   , piece_id=IEHPTP4N7477FC3JV4ZWUHTZTBWEO5VOXOHDO7YIO46BM3EKKCRQ, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:25.449125056 (587h55m25.320208101s) [noise=true , address=77.90.31.16:4735     , piece_id=WR6T6EJTAOENHFF5IGSM6NTI5GBMHANLCCL5XGKXCKV4OBU2GXQA, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:25.472119190 (587h55m25.343202235s) [noise=true , address=77.90.31.19:4844     , piece_id=KOLINXXA6TQ62BCUVZ2R26C5HLBBDORJKKUSC2B746WBDPXVIAGA, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:25.522011752 (587h55m25.393094787s) [noise=true , address=45.91.116.7:4011     , piece_id=3PNYKM4ZZUET4CUAEGS47NHOCG547ESP6PNAKYUPGB5K6DF3LUNQ, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:26.010620850 (587h55m25.881703895s) [noise=true , address=77.90.31.10:4520     , piece_id=PWGM5QHD7NEXABJK5BLRWMVW26BFKFADAKGEGZ2CVPGQIKWVP4VA, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:26.034576585 (587h55m25.90565962s) [noise=true , address=77.90.31.17:4773     , piece_id=JICWFBYEC7EGLMT5K6CC25AJEPLK5WLXMO2MOHST435YN56ZMECQ, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:26.156531354 (587h55m26.027614399s) [noise=true , address=45.91.116.12:4225    , piece_id=2EZ6PVGQ3U4AKFFXA7R7IWEHX7LDBGOAOXBQIRMCDBB3FII5W7QA, seg_pos={0 0}, upload=515382]: Done uploading piece. err: <nil>
Dec 30 03:24:26.156552233 (587h55m26.027635268s) [seg_pos={0 0}, upload=515382]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:24:26.156754699 (587h55m26.027837744s) [noise=true , address=145.79.227.37:4565   , piece_id=XSSZN3A7BCL77K72MJ3BZ2VPUBLAQYA5BOZX5JA2W3U4B2EWYPEQ, seg_pos={0 0}, upload=515382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:42114->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:42114->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42114->145.79.227.37:4565: use of closed network connection
Dec 30 03:24:26.156760770 (587h55m26.027843805s) [noise=true , address=145.79.227.37:4565   , piece_id=XSSZN3A7BCL77K72MJ3BZ2VPUBLAQYA5BOZX5JA2W3U4B2EWYPEQ, seg_pos={0 0}, upload=515382]: Long tail cancellation due to optimal threshold
Dec 30 03:24:26.156828740 (587h55m26.027911795s) [noise=true , address=145.79.227.50:4374   , piece_id=TZQTA3K6K3MTL4RE4O4GFOGNHQGAGQ2OSA23GVX2D7I5NBDXC7YA, seg_pos={0 0}, upload=515382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: write tcp 188.166.225.170:40760->145.79.227.50:4374: use of closed network connection; write tcp 188.166.225.170:40760->145.79.227.50:4374: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40760->145.79.227.50:4374: use of closed network connection
Dec 30 03:24:26.156862083 (587h55m26.027945118s) [noise=true , address=145.79.227.50:4374   , piece_id=TZQTA3K6K3MTL4RE4O4GFOGNHQGAGQ2OSA23GVX2D7I5NBDXC7YA, seg_pos={0 0}, upload=515382]: Long tail cancellation due to optimal threshold
Dec 30 03:24:26.156892942 (587h55m26.027975977s) [noise=true , address=145.79.227.34:4906   , piece_id=P7R2FYPDYLWZUPD32JI2KYP3DNQMZOQ5HQGAC2D7DJBXIZ3MHCNA, seg_pos={0 0}, upload=515382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 30 03:24:26.156899454 (587h55m26.027982499s) [noise=true , address=145.79.227.34:4906   , piece_id=P7R2FYPDYLWZUPD32JI2KYP3DNQMZOQ5HQGAC2D7DJBXIZ3MHCNA, seg_pos={0 0}, upload=515382]: Long tail cancellation due to optimal threshold
Dec 30 03:24:26.156916197 (587h55m26.027999242s) [seg_pos={0 0}, upload=515382]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:24:26.156981952 (587h55m26.028064987s) [upload=515382]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:24:26.156985779 (587h55m26.028068814s) [upload=515382]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:24:26.453407748 (587h55m26.324490783s) [upload=515382]: Done uploading segments...
Dec 30 03:24:26.453425722 (587h55m26.324508757s) [upload=515382]: Upload finished. err: <nil>
Duration: 30.002945201s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [........X...X.X.XX................XX............X....X.X....]
duration: 
4.147927253s
trace: 
https://jaeger.dev.storj.io/trace/476bb7d811b386c5
FAIL (2025-12-30 03:24:41 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.147960766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [................X......X....................................]
duration: 
41.799842532s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:24:35 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 41.79986245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [.........X...X.X.XX................XX............X....X.X...]
duration: 
3.397722509s
trace: 
https://jaeger.dev.storj.io/trace/476bb7d811b386c5
FAIL (2025-12-30 03:24:12 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.397753779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [X................X......X...................................]
duration: 
41.361903338s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:23:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 41.361933828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [..........X...X.X.XX................XX............X....X.X..]
duration: 
5.000763304s
trace: 
https://jaeger.dev.storj.io/trace/476bb7d811b386c5
FAIL (2025-12-30 03:23:52 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.000789825s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [...........X...X.X.XX................XX............X....X.X.]
duration: 
4.340890377s
trace: 
https://jaeger.dev.storj.io/trace/476bb7d811b386c5
FAIL (2025-12-30 03:23:36 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.340917599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X.X...XXX.]
duration: 
6.955249009s
trace: 
https://jaeger.dev.storj.io/trace/59adf357f5791c10
FAIL (2025-12-30 03:23:21 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: PQD2AWYQWQ6MVRYNUVFAXR4K2BWWZTW42HF3K4UZ7W5QYTTBM5HA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ZF5JA5OYNBUWPG44ZKFHA7K2O45T7WLTN26XBPYJVR6ZX3TKCQKA) context deadline exceeded
======LOGS======
3:22:10.663887716 (587h53m10.534970762s) [upload=515353]: Starting upload
Dec 30 03:22:10.663999579 (587h53m10.535082624s) [upload=515353]: Done starting upload
Dec 30 03:22:10.664086225 (587h53m10.53516927s) [upload=515353]: Uploading segments...
Dec 30 03:22:10.664249755 (587h53m10.535332801s) [upload=515353]: Got next segment. Inline: false
Dec 30 03:22:10.664290965 (587h53m10.535374s) [upload=515353]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:22:10.664297988 (587h53m10.535381033s) [upload=515353]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:22:10.914778497 (587h53m10.785861542s) [seg_pos={0 0}, upload=515353]: Begin upload segment...
Dec 30 03:22:10.914842218 (587h53m10.785925263s) [seg_pos={0 0}, upload=515353]: Done begin upload segment.
Dec 30 03:22:10.914889077 (587h53m10.785972122s) [noise=true , address=145.79.227.18:4006   , piece_id=LI3UD7I2ZUM7ASTTADI2UWCZFSCSBW5MMXVFAYOWQN5YAWW52WUQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.914915327 (587h53m10.785998362s) [noise=true , address=45.91.116.5:4297     , piece_id=PQD2AWYQWQ6MVRYNUVFAXR4K2BWWZTW42HF3K4UZ7W5QYTTBM5HA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.915487737 (587h53m10.786570782s) [noise=true , address=145.79.227.15:4874   , piece_id=5YSOQCV2IYJ62BJV3Z4EY4N2TDGLHUAQ6PESMGAEZ7YB2R2EIGRQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.915780174 (587h53m10.786863219s) [noise=true , address=145.79.227.38:4978   , piece_id=4T4QJ4IIZT4RFJFPJKWD7PYBCJAMZBT77S4ZWAHYFNI34WHQPZPA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.916157964 (587h53m10.787241009s) [noise=true , address=45.91.116.14:4192    , piece_id=RVSGL6X233262GNZHF57FMVT4675EJ6ZM5SUIGHCVKZXFVDDBE3A, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.916475338 (587h53m10.787558373s) [noise=true , address=145.79.227.3:4149    , piece_id=STJPSQFTJJFMOT5W2TZDLCZPXFSPOPQ4WZHAAP2JJ4RUE7YCWGTA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.916730103 (587h53m10.787813138s) [noise=true , address=145.79.227.51:4409   , piece_id=OKAZKP3EX23PCWR7B7P2GI2NOVUHBVZM7KRZY7FGHBKUEXBC64KQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.916787763 (587h53m10.787870808s) [noise=true , address=77.90.31.11:4555     , piece_id=3775WUU4DW75N6Q5GMQ4IA7JXK426BWZKTI7UHQLZ2MYUU7TECHA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.917102011 (587h53m10.788185056s) [noise=true , address=145.79.227.17:4971   , piece_id=SIKRZB6CNRIRDV4YPLSXPKRWBT4BSVKDCV4D5TN4CYW5SCW7ZSYA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.917439563 (587h53m10.788522608s) [noise=true , address=145.79.227.29:4943   , piece_id=RZYGP4QX422AAKYF6LJIOGLOCDLKPVKUGCJZYY7ROTSEA35ZTZPQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.917997426 (587h53m10.789080472s) [noise=true , address=145.79.227.45:4231   , piece_id=HLKVBXM3GT5D26RML2IP2C4FIHZRDSQAQLEK3QGOSZXXJ7332HVQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.918462802 (587h53m10.789545837s) [noise=true , address=45.91.116.6:4975     , piece_id=ZF5JA5OYNBUWPG44ZKFHA7K2O45T7WLTN26XBPYJVR6ZX3TKCQKA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.918844800 (587h53m10.789927845s) [noise=true , address=145.79.227.52:4447   , piece_id=3JANG4YY6QW3XF352RJEZF3SBXHPX6D3Y2BZQRCYZPYSNQCWPTJQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.919221327 (587h53m10.790304372s) [noise=true , address=145.79.227.10:4401   , piece_id=VK7JMDN3HDRJA6NAXZHPZ4SZGY4APSSPXWED4FOVCWMSNN2B7T7A, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.919615948 (587h53m10.790698993s) [noise=true , address=145.79.227.32:4836   , piece_id=BL5E2JR5YIEZECGTQUUBYMLPVJR7P6I5XYZTDOC7KTLIUYXU63UQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.920207855 (587h53m10.7912909s) [noise=true , address=145.79.227.39:4013   , piece_id=OZ4BJ22GZ4OIAT6N4M55GB6PXCJ6CMI7PCLUQJJTQDUWKGKZC4IQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:12.612404340 (587h53m12.483487375s) [upload=515353]: Next returned nil segment
Dec 30 03:22:12.612421834 (587h53m12.483504869s) [upload=515353]: Waiting for error group managing segments...
Dec 30 03:22:12.885640660 (587h53m12.756723695s) [noise=true , address=145.79.227.15:4874   , piece_id=5YSOQCV2IYJ62BJV3Z4EY4N2TDGLHUAQ6PESMGAEZ7YB2R2EIGRQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.891942820 (587h53m12.763025865s) [noise=true , address=145.79.227.3:4149    , piece_id=STJPSQFTJJFMOT5W2TZDLCZPXFSPOPQ4WZHAAP2JJ4RUE7YCWGTA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.905684176 (587h53m12.776767211s) [noise=true , address=145.79.227.18:4006   , piece_id=LI3UD7I2ZUM7ASTTADI2UWCZFSCSBW5MMXVFAYOWQN5YAWW52WUQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.914428777 (587h53m12.785511812s) [noise=true , address=145.79.227.51:4409   , piece_id=OKAZKP3EX23PCWR7B7P2GI2NOVUHBVZM7KRZY7FGHBKUEXBC64KQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.914583151 (587h53m12.785666196s) [noise=true , address=145.79.227.39:4013   , piece_id=OZ4BJ22GZ4OIAT6N4M55GB6PXCJ6CMI7PCLUQJJTQDUWKGKZC4IQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.927429802 (587h53m12.798512837s) [noise=true , address=145.79.227.45:4231   , piece_id=HLKVBXM3GT5D26RML2IP2C4FIHZRDSQAQLEK3QGOSZXXJ7332HVQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.928010739 (587h53m12.799093784s) [noise=true , address=145.79.227.10:4401   , piece_id=VK7JMDN3HDRJA6NAXZHPZ4SZGY4APSSPXWED4FOVCWMSNN2B7T7A, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.931451350 (587h53m12.802534386s) [noise=true , address=145.79.227.32:4836   , piece_id=BL5E2JR5YIEZECGTQUUBYMLPVJR7P6I5XYZTDOC7KTLIUYXU63UQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.935785613 (587h53m12.806868648s) [noise=true , address=145.79.227.38:4978   , piece_id=4T4QJ4IIZT4RFJFPJKWD7PYBCJAMZBT77S4ZWAHYFNI34WHQPZPA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.940325397 (587h53m12.811408432s) [noise=true , address=145.79.227.52:4447   , piece_id=3JANG4YY6QW3XF352RJEZF3SBXHPX6D3Y2BZQRCYZPYSNQCWPTJQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:13.147966625 (587h53m13.01904967s) [noise=true , address=45.91.116.5:4297     , piece_id=PQD2AWYQWQ6MVRYNUVFAXR4K2BWWZTW42HF3K4UZ7W5QYTTBM5HA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:13.222978078 (587h53m13.094061113s) [noise=true , address=45.91.116.6:4975     , piece_id=ZF5JA5OYNBUWPG44ZKFHA7K2O45T7WLTN26XBPYJVR6ZX3TKCQKA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:13.840275898 (587h53m13.711358973s) [noise=true , address=77.90.31.11:4555     , piece_id=3775WUU4DW75N6Q5GMQ4IA7JXK426BWZKTI7UHQLZ2MYUU7TECHA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:13.840306306 (587h53m13.711389341s) [seg_pos={0 0}, upload=515353]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:22:13.840502640 (587h53m13.711585675s) [noise=true , address=145.79.227.17:4971   , piece_id=SIKRZB6CNRIRDV4YPLSXPKRWBT4BSVKDCV4D5TN4CYW5SCW7ZSYA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:58652->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:58652->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->145.79.227.17:4971: use of closed network connection
Dec 30 03:22:13.840509052 (587h53m13.711592087s) [noise=true , address=145.79.227.17:4971   , piece_id=SIKRZB6CNRIRDV4YPLSXPKRWBT4BSVKDCV4D5TN4CYW5SCW7ZSYA, seg_pos={0 0}, upload=515353]: Long tail cancellation due to optimal threshold
Dec 30 03:22:13.840701339 (587h53m13.711784374s) [noise=true , address=45.91.116.14:4192    , piece_id=RVSGL6X233262GNZHF57FMVT4675EJ6ZM5SUIGHCVKZXFVDDBE3A, seg_pos={0 0}, upload=515353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 30 03:22:13.840706959 (587h53m13.711789994s) [noise=true , address=45.91.116.14:4192    , piece_id=RVSGL6X233262GNZHF57FMVT4675EJ6ZM5SUIGHCVKZXFVDDBE3A, seg_pos={0 0}, upload=515353]: Long tail cancellation due to optimal threshold
Dec 30 03:22:13.840748318 (587h53m13.711831373s) [noise=true , address=145.79.227.29:4943   , piece_id=RZYGP4QX422AAKYF6LJIOGLOCDLKPVKUGCJZYY7ROTSEA35ZTZPQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 30 03:22:13.840769018 (587h53m13.711852073s) [noise=true , address=145.79.227.29:4943   , piece_id=RZYGP4QX422AAKYF6LJIOGLOCDLKPVKUGCJZYY7ROTSEA35ZTZPQ, seg_pos={0 0}, upload=515353]: Long tail cancellation due to optimal threshold
Dec 30 03:22:13.840796459 (587h53m13.711879494s) [seg_pos={0 0}, upload=515353]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:22:13.840866693 (587h53m13.711949738s) [upload=515353]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:22:13.840871082 (587h53m13.711954117s) [upload=515353]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:22:14.143967969 (587h53m14.015051014s) [upload=515353]: Done uploading segments...
Dec 30 03:22:14.143990853 (587h53m14.015073888s) [upload=515353]: Upload finished. err: <nil>
Duration: 6.955276191s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [............X...X.X.XX................XX............X....X.X]
duration: 
5.000941025s
trace: 
https://jaeger.dev.storj.io/trace/476bb7d811b386c5
FAIL (2025-12-30 03:23:03 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.000967766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [X..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X.X...XXX]
duration: 
30.00408773s
trace: 
https://jaeger.dev.storj.io/trace/59adf357f5791c10
FAIL (2025-12-30 03:22:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: PQD2AWYQWQ6MVRYNUVFAXR4K2BWWZTW42HF3K4UZ7W5QYTTBM5HA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ZF5JA5OYNBUWPG44ZKFHA7K2O45T7WLTN26XBPYJVR6ZX3TKCQKA) context deadline exceeded
======LOGS======
3:22:10.663887716 (587h53m10.534970762s) [upload=515353]: Starting upload
Dec 30 03:22:10.663999579 (587h53m10.535082624s) [upload=515353]: Done starting upload
Dec 30 03:22:10.664086225 (587h53m10.53516927s) [upload=515353]: Uploading segments...
Dec 30 03:22:10.664249755 (587h53m10.535332801s) [upload=515353]: Got next segment. Inline: false
Dec 30 03:22:10.664290965 (587h53m10.535374s) [upload=515353]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:22:10.664297988 (587h53m10.535381033s) [upload=515353]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:22:10.914778497 (587h53m10.785861542s) [seg_pos={0 0}, upload=515353]: Begin upload segment...
Dec 30 03:22:10.914842218 (587h53m10.785925263s) [seg_pos={0 0}, upload=515353]: Done begin upload segment.
Dec 30 03:22:10.914889077 (587h53m10.785972122s) [noise=true , address=145.79.227.18:4006   , piece_id=LI3UD7I2ZUM7ASTTADI2UWCZFSCSBW5MMXVFAYOWQN5YAWW52WUQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.914915327 (587h53m10.785998362s) [noise=true , address=45.91.116.5:4297     , piece_id=PQD2AWYQWQ6MVRYNUVFAXR4K2BWWZTW42HF3K4UZ7W5QYTTBM5HA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.915487737 (587h53m10.786570782s) [noise=true , address=145.79.227.15:4874   , piece_id=5YSOQCV2IYJ62BJV3Z4EY4N2TDGLHUAQ6PESMGAEZ7YB2R2EIGRQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.915780174 (587h53m10.786863219s) [noise=true , address=145.79.227.38:4978   , piece_id=4T4QJ4IIZT4RFJFPJKWD7PYBCJAMZBT77S4ZWAHYFNI34WHQPZPA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.916157964 (587h53m10.787241009s) [noise=true , address=45.91.116.14:4192    , piece_id=RVSGL6X233262GNZHF57FMVT4675EJ6ZM5SUIGHCVKZXFVDDBE3A, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.916475338 (587h53m10.787558373s) [noise=true , address=145.79.227.3:4149    , piece_id=STJPSQFTJJFMOT5W2TZDLCZPXFSPOPQ4WZHAAP2JJ4RUE7YCWGTA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.916730103 (587h53m10.787813138s) [noise=true , address=145.79.227.51:4409   , piece_id=OKAZKP3EX23PCWR7B7P2GI2NOVUHBVZM7KRZY7FGHBKUEXBC64KQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.916787763 (587h53m10.787870808s) [noise=true , address=77.90.31.11:4555     , piece_id=3775WUU4DW75N6Q5GMQ4IA7JXK426BWZKTI7UHQLZ2MYUU7TECHA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.917102011 (587h53m10.788185056s) [noise=true , address=145.79.227.17:4971   , piece_id=SIKRZB6CNRIRDV4YPLSXPKRWBT4BSVKDCV4D5TN4CYW5SCW7ZSYA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.917439563 (587h53m10.788522608s) [noise=true , address=145.79.227.29:4943   , piece_id=RZYGP4QX422AAKYF6LJIOGLOCDLKPVKUGCJZYY7ROTSEA35ZTZPQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.917997426 (587h53m10.789080472s) [noise=true , address=145.79.227.45:4231   , piece_id=HLKVBXM3GT5D26RML2IP2C4FIHZRDSQAQLEK3QGOSZXXJ7332HVQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.918462802 (587h53m10.789545837s) [noise=true , address=45.91.116.6:4975     , piece_id=ZF5JA5OYNBUWPG44ZKFHA7K2O45T7WLTN26XBPYJVR6ZX3TKCQKA, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.918844800 (587h53m10.789927845s) [noise=true , address=145.79.227.52:4447   , piece_id=3JANG4YY6QW3XF352RJEZF3SBXHPX6D3Y2BZQRCYZPYSNQCWPTJQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.919221327 (587h53m10.790304372s) [noise=true , address=145.79.227.10:4401   , piece_id=VK7JMDN3HDRJA6NAXZHPZ4SZGY4APSSPXWED4FOVCWMSNN2B7T7A, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.919615948 (587h53m10.790698993s) [noise=true , address=145.79.227.32:4836   , piece_id=BL5E2JR5YIEZECGTQUUBYMLPVJR7P6I5XYZTDOC7KTLIUYXU63UQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:10.920207855 (587h53m10.7912909s) [noise=true , address=145.79.227.39:4013   , piece_id=OZ4BJ22GZ4OIAT6N4M55GB6PXCJ6CMI7PCLUQJJTQDUWKGKZC4IQ, seg_pos={0 0}, upload=515353]: Uploading piece...
Dec 30 03:22:12.612404340 (587h53m12.483487375s) [upload=515353]: Next returned nil segment
Dec 30 03:22:12.612421834 (587h53m12.483504869s) [upload=515353]: Waiting for error group managing segments...
Dec 30 03:22:12.885640660 (587h53m12.756723695s) [noise=true , address=145.79.227.15:4874   , piece_id=5YSOQCV2IYJ62BJV3Z4EY4N2TDGLHUAQ6PESMGAEZ7YB2R2EIGRQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.891942820 (587h53m12.763025865s) [noise=true , address=145.79.227.3:4149    , piece_id=STJPSQFTJJFMOT5W2TZDLCZPXFSPOPQ4WZHAAP2JJ4RUE7YCWGTA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.905684176 (587h53m12.776767211s) [noise=true , address=145.79.227.18:4006   , piece_id=LI3UD7I2ZUM7ASTTADI2UWCZFSCSBW5MMXVFAYOWQN5YAWW52WUQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.914428777 (587h53m12.785511812s) [noise=true , address=145.79.227.51:4409   , piece_id=OKAZKP3EX23PCWR7B7P2GI2NOVUHBVZM7KRZY7FGHBKUEXBC64KQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.914583151 (587h53m12.785666196s) [noise=true , address=145.79.227.39:4013   , piece_id=OZ4BJ22GZ4OIAT6N4M55GB6PXCJ6CMI7PCLUQJJTQDUWKGKZC4IQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.927429802 (587h53m12.798512837s) [noise=true , address=145.79.227.45:4231   , piece_id=HLKVBXM3GT5D26RML2IP2C4FIHZRDSQAQLEK3QGOSZXXJ7332HVQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.928010739 (587h53m12.799093784s) [noise=true , address=145.79.227.10:4401   , piece_id=VK7JMDN3HDRJA6NAXZHPZ4SZGY4APSSPXWED4FOVCWMSNN2B7T7A, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.931451350 (587h53m12.802534386s) [noise=true , address=145.79.227.32:4836   , piece_id=BL5E2JR5YIEZECGTQUUBYMLPVJR7P6I5XYZTDOC7KTLIUYXU63UQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.935785613 (587h53m12.806868648s) [noise=true , address=145.79.227.38:4978   , piece_id=4T4QJ4IIZT4RFJFPJKWD7PYBCJAMZBT77S4ZWAHYFNI34WHQPZPA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:12.940325397 (587h53m12.811408432s) [noise=true , address=145.79.227.52:4447   , piece_id=3JANG4YY6QW3XF352RJEZF3SBXHPX6D3Y2BZQRCYZPYSNQCWPTJQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:13.147966625 (587h53m13.01904967s) [noise=true , address=45.91.116.5:4297     , piece_id=PQD2AWYQWQ6MVRYNUVFAXR4K2BWWZTW42HF3K4UZ7W5QYTTBM5HA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:13.222978078 (587h53m13.094061113s) [noise=true , address=45.91.116.6:4975     , piece_id=ZF5JA5OYNBUWPG44ZKFHA7K2O45T7WLTN26XBPYJVR6ZX3TKCQKA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:13.840275898 (587h53m13.711358973s) [noise=true , address=77.90.31.11:4555     , piece_id=3775WUU4DW75N6Q5GMQ4IA7JXK426BWZKTI7UHQLZ2MYUU7TECHA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: <nil>
Dec 30 03:22:13.840306306 (587h53m13.711389341s) [seg_pos={0 0}, upload=515353]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:22:13.840502640 (587h53m13.711585675s) [noise=true , address=145.79.227.17:4971   , piece_id=SIKRZB6CNRIRDV4YPLSXPKRWBT4BSVKDCV4D5TN4CYW5SCW7ZSYA, seg_pos={0 0}, upload=515353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:58652->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:58652->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->145.79.227.17:4971: use of closed network connection
Dec 30 03:22:13.840509052 (587h53m13.711592087s) [noise=true , address=145.79.227.17:4971   , piece_id=SIKRZB6CNRIRDV4YPLSXPKRWBT4BSVKDCV4D5TN4CYW5SCW7ZSYA, seg_pos={0 0}, upload=515353]: Long tail cancellation due to optimal threshold
Dec 30 03:22:13.840701339 (587h53m13.711784374s) [noise=true , address=45.91.116.14:4192    , piece_id=RVSGL6X233262GNZHF57FMVT4675EJ6ZM5SUIGHCVKZXFVDDBE3A, seg_pos={0 0}, upload=515353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 30 03:22:13.840706959 (587h53m13.711789994s) [noise=true , address=45.91.116.14:4192    , piece_id=RVSGL6X233262GNZHF57FMVT4675EJ6ZM5SUIGHCVKZXFVDDBE3A, seg_pos={0 0}, upload=515353]: Long tail cancellation due to optimal threshold
Dec 30 03:22:13.840748318 (587h53m13.711831373s) [noise=true , address=145.79.227.29:4943   , piece_id=RZYGP4QX422AAKYF6LJIOGLOCDLKPVKUGCJZYY7ROTSEA35ZTZPQ, seg_pos={0 0}, upload=515353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 30 03:22:13.840769018 (587h53m13.711852073s) [noise=true , address=145.79.227.29:4943   , piece_id=RZYGP4QX422AAKYF6LJIOGLOCDLKPVKUGCJZYY7ROTSEA35ZTZPQ, seg_pos={0 0}, upload=515353]: Long tail cancellation due to optimal threshold
Dec 30 03:22:13.840796459 (587h53m13.711879494s) [seg_pos={0 0}, upload=515353]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:22:13.840866693 (587h53m13.711949738s) [upload=515353]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:22:13.840871082 (587h53m13.711954117s) [upload=515353]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:22:14.143967969 (587h53m14.015051014s) [upload=515353]: Done uploading segments...
Dec 30 03:22:14.143990853 (587h53m14.015073888s) [upload=515353]: Upload finished. err: <nil>
Duration: 30.004100714s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [.............X...X.X.XX................XX............X....X.]
duration: 
3.506231408s
trace: 
https://jaeger.dev.storj.io/trace/3e7b9c4ee3928be6
FAIL (2025-12-30 03:22:33 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.506267166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.X................X......X..................................]
duration: 
21.910954194s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:22:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 21.910986075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X.X...XX]
duration: 
30.002879828s
trace: 
https://jaeger.dev.storj.io/trace/523700fcec39569a
FAIL (2025-12-30 03:22:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.
Dec 30 03:21:40.412857996 (587h52m40.283941042s) [upload=515342]: Got next segment. Inline: false
Dec 30 03:21:40.412886780 (587h52m40.283969826s) [upload=515342]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:21:40.412891320 (587h52m40.283974365s) [upload=515342]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:21:40.656377306 (587h52m40.527460361s) [seg_pos={0 0}, upload=515342]: Begin upload segment...
Dec 30 03:21:40.656444394 (587h52m40.527527439s) [seg_pos={0 0}, upload=515342]: Done begin upload segment.
Dec 30 03:21:40.656504287 (587h52m40.527587322s) [noise=true , address=145.79.227.8:4331    , piece_id=KG2CX5TNNZDQN2B2OHRZTHECTZUMXSWHAR6B44OSVUTL7TNI3JMQ, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.656521230 (587h52m40.527604265s) [noise=true , address=45.91.116.7:4011     , piece_id=CWARSF5RU5XUL62VB2RDFCA3XDZPGCLWGPV53VPLRYWPQ4TVC6SQ, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.657208929 (587h52m40.528291964s) [noise=true , address=45.91.116.11:4117    , piece_id=UPEDXFJ4TG5I7PH32EAMFUY73BLLDMMIJG3E6ZDXOPGQXC3B6YKQ, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.657318919 (587h52m40.528401954s) [noise=true , address=145.79.227.15:4873   , piece_id=5MEYJ6MEDZFH35ISCXNA2KHJOEXCP2WJA6ZQRMYJCDOMF4XQZONQ, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.657589344 (587h52m40.528672389s) [noise=true , address=145.79.227.26:4654   , piece_id=6ZBL6PABU7R3LUQBSSTTRU2R2BNNJ5CJZTUP4JM47ZSN36S7PLCA, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.657885988 (587h52m40.528969033s) [noise=true , address=145.79.227.16:4911   , piece_id=JPPVBOMLKZNNCCUUVCFK3DII73BROULRKBODCJPGQ3SLSBC3UROA, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.658015946 (587h52m40.529098981s) [noise=true , address=45.91.116.14:4191    , piece_id=TSPOEMQ3UJCTPGM6RPLN2J2WJJNLQW7MEOUCKDZTBWAIEJVN3KHA, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.658474820 (587h52m40.529557855s) [noise=true , address=145.79.227.17:4971   , piece_id=M5TU7OSHJOJ6VL66MUO63U74J2TS3M64JJD5JAKNJJXPCTQTREXA, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.658561315 (587h52m40.52964435s) [noise=true , address=45.91.116.12:4227    , piece_id=DLMDB6PAYFTA7HLP65HGRWD5R2A2MZGTAWS4KETNNLY3EYQNALIQ, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.658943873 (587h52m40.530026918s) [noise=true , address=145.79.227.10:4402   , piece_id=4XJVE5DEMSPW42VMN2NL47D2EGEPO7NYK347L7KQLUDUZNRWQLXQ, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.659433275 (587h52m40.53051631s) [noise=true , address=145.79.227.21:4475   , piece_id=UPPAEFL4D55YISPOCXQRZLNEB25XSXFY7GNK3YQBQVQRO32VZBKA, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.659802578 (587h52m40.530885623s) [noise=true , address=145.79.227.29:4943   , piece_id=IFXCCYXSCDYQRL56CUUBKSOXDRSGHSMQL4QSRFL3ZJKZFSJLOQCQ, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.659862001 (587h52m40.530945036s) [noise=true , address=145.79.227.5:4222    , piece_id=4WFZBLBDKVNG3FO3ZB2HETUEKGGX6F6ORNSLTC4OGMMK2DBN6B6Q, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.660302180 (587h52m40.531385225s) [noise=true , address=145.79.227.11:4730   , piece_id=GRNQGZBZGG5BXZWM6KRFHPG27U7GRQWOZLTAFQKOVI5NL4AAXXSQ, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.660691771 (587h52m40.531774806s) [noise=true , address=45.91.116.4:4939     , piece_id=VOHLTCXOED6S3ATUJPZVDKCOBZNJALD3WDFD4SFTRJDVXUSV67EA, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:40.660929815 (587h52m40.53201287s) [noise=true , address=145.79.227.42:4123   , piece_id=JUKOZSTOKAL6SZF54V657AABFVELHGIIHLNPUB4FP5LSXGH7DJLA, seg_pos={0 0}, upload=515342]: Uploading piece...
Dec 30 03:21:42.359071143 (587h52m42.230154178s) [upload=515342]: Next returned nil segment
Dec 30 03:21:42.359084558 (587h52m42.230167603s) [upload=515342]: Waiting for error group managing segments...
Dec 30 03:21:42.641351862 (587h52m42.512434907s) [noise=true , address=145.79.227.16:4911   , piece_id=JPPVBOMLKZNNCCUUVCFK3DII73BROULRKBODCJPGQ3SLSBC3UROA, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:42.641769117 (587h52m42.512852162s) [noise=true , address=145.79.227.8:4331    , piece_id=KG2CX5TNNZDQN2B2OHRZTHECTZUMXSWHAR6B44OSVUTL7TNI3JMQ, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:42.642493134 (587h52m42.51357618s) [noise=true , address=145.79.227.5:4222    , piece_id=4WFZBLBDKVNG3FO3ZB2HETUEKGGX6F6ORNSLTC4OGMMK2DBN6B6Q, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:42.648688431 (587h52m42.519771477s) [noise=true , address=145.79.227.15:4873   , piece_id=5MEYJ6MEDZFH35ISCXNA2KHJOEXCP2WJA6ZQRMYJCDOMF4XQZONQ, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:42.653516054 (587h52m42.524599099s) [noise=true , address=145.79.227.21:4475   , piece_id=UPPAEFL4D55YISPOCXQRZLNEB25XSXFY7GNK3YQBQVQRO32VZBKA, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:42.660169804 (587h52m42.531252839s) [noise=true , address=145.79.227.10:4402   , piece_id=4XJVE5DEMSPW42VMN2NL47D2EGEPO7NYK347L7KQLUDUZNRWQLXQ, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:42.665714151 (587h52m42.536797186s) [noise=true , address=145.79.227.11:4730   , piece_id=GRNQGZBZGG5BXZWM6KRFHPG27U7GRQWOZLTAFQKOVI5NL4AAXXSQ, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:42.709628642 (587h52m42.580711687s) [noise=true , address=145.79.227.42:4123   , piece_id=JUKOZSTOKAL6SZF54V657AABFVELHGIIHLNPUB4FP5LSXGH7DJLA, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:42.910938268 (587h52m42.782021303s) [noise=true , address=45.91.116.7:4011     , piece_id=CWARSF5RU5XUL62VB2RDFCA3XDZPGCLWGPV53VPLRYWPQ4TVC6SQ, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:42.930376672 (587h52m42.801459708s) [noise=true , address=45.91.116.11:4117    , piece_id=UPEDXFJ4TG5I7PH32EAMFUY73BLLDMMIJG3E6ZDXOPGQXC3B6YKQ, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:42.932938821 (587h52m42.804021856s) [noise=true , address=45.91.116.12:4227    , piece_id=DLMDB6PAYFTA7HLP65HGRWD5R2A2MZGTAWS4KETNNLY3EYQNALIQ, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:43.300122428 (587h52m43.171205473s) [noise=true , address=45.91.116.4:4939     , piece_id=VOHLTCXOED6S3ATUJPZVDKCOBZNJALD3WDFD4SFTRJDVXUSV67EA, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:43.666790634 (587h52m43.537873679s) [noise=true , address=45.91.116.14:4191    , piece_id=TSPOEMQ3UJCTPGM6RPLN2J2WJJNLQW7MEOUCKDZTBWAIEJVN3KHA, seg_pos={0 0}, upload=515342]: Done uploading piece. err: <nil>
Dec 30 03:21:43.666812837 (587h52m43.537895882s) [seg_pos={0 0}, upload=515342]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:21:43.667016976 (587h52m43.538100011s) [noise=true , address=145.79.227.17:4971   , piece_id=M5TU7OSHJOJ6VL66MUO63U74J2TS3M64JJD5JAKNJJXPCTQTREXA, seg_pos={0 0}, upload=515342]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:50174->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:50174->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50174->145.79.227.17:4971: use of closed network connection
Dec 30 03:21:43.667023067 (587h52m43.538106102s) [noise=true , address=145.79.227.17:4971   , piece_id=M5TU7OSHJOJ6VL66MUO63U74J2TS3M64JJD5JAKNJJXPCTQTREXA, seg_pos={0 0}, upload=515342]: Long tail cancellation due to optimal threshold
Dec 30 03:21:43.667262232 (587h52m43.538345268s) [noise=true , address=145.79.227.29:4943   , piece_id=IFXCCYXSCDYQRL56CUUBKSOXDRSGHSMQL4QSRFL3ZJKZFSJLOQCQ, seg_pos={0 0}, upload=515342]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 30 03:21:43.667270197 (587h52m43.538353232s) [noise=true , address=145.79.227.29:4943   , piece_id=IFXCCYXSCDYQRL56CUUBKSOXDRSGHSMQL4QSRFL3ZJKZFSJLOQCQ, seg_pos={0 0}, upload=515342]: Long tail cancellation due to optimal threshold
Dec 30 03:21:43.667274996 (587h52m43.538358031s) [noise=true , address=145.79.227.26:4654   , piece_id=6ZBL6PABU7R3LUQBSSTTRU2R2BNNJ5CJZTUP4JM47ZSN36S7PLCA, seg_pos={0 0}, upload=515342]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:43948->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:43948->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43948->145.79.227.26:4654: use of closed network connection
Dec 30 03:21:43.667289715 (587h52m43.53837276s) [noise=true , address=145.79.227.26:4654   , piece_id=6ZBL6PABU7R3LUQBSSTTRU2R2BNNJ5CJZTUP4JM47ZSN36S7PLCA, seg_pos={0 0}, upload=515342]: Long tail cancellation due to optimal threshold
Dec 30 03:21:43.667314001 (587h52m43.538397036s) [seg_pos={0 0}, upload=515342]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:21:43.667496399 (587h52m43.538579434s) [upload=515342]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:21:43.667502911 (587h52m43.538585946s) [upload=515342]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:21:43.999315285 (587h52m43.87039832s) [upload=515342]: Done uploading segments...
Dec 30 03:21:43.999336436 (587h52m43.870419481s) [upload=515342]: Upload finished. err: <nil>
Duration: 30.002903032s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [..............X...X.X.XX................XX............X....X]
duration: 
5.001277038s
trace: 
https://jaeger.dev.storj.io/trace/3e7b9c4ee3928be6
FAIL (2025-12-30 03:21:45 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001311616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..X................X......X.................................]
duration: 
37.645427524s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:21:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 37.645444396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [.XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X.X...X]
duration: 
30.002535811s
trace: 
https://jaeger.dev.storj.io/trace/1e106005c7404994
FAIL (2025-12-30 03:21:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 3K7P6VSFLLHDK5SSVL7CAW266M35ZIPJMKM47SDTZDQNZKLWQ3FA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: SIGRDBV7INJ2VB4RBHVENZBZAGSHUHSBWV23ABQET3ILLHMVOSHQ) context deadline exceeded
======LOGS======
Dec 30 03:20:39.768301436 (587h51m39.639384472s) [upload=515329]: Starting upload
Dec 30 03:20:39.768358735 (587h51m39.63944177s) [upload=515329]: Done starting upload
Dec 30 03:20:39.768483062 (587h51m39.639566107s) [upload=515329]: Uploading segments...
Dec 30 03:20:39.768576310 (587h51m39.639659355s) [upload=515329]: Got next segment. Inline: false
Dec 30 03:20:39.768630152 (587h51m39.639713197s) [upload=515329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:20:39.768637025 (587h51m39.63972007s) [upload=515329]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:20:40.014914688 (587h51m39.885997733s) [seg_pos={0 0}, upload=515329]: Begin upload segment...
Dec 30 03:20:40.014969834 (587h51m39.886052869s) [seg_pos={0 0}, upload=515329]: Done begin upload segment.
Dec 30 03:20:40.015000823 (587h51m39.886083868s) [noise=true , address=77.90.31.19:4844     , piece_id=EKAGFL7AEGC3AMPDGQBAFWTC7NOD5N7CGWCNBJGTC5PQC4T34GFA, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.015073431 (587h51m39.886156466s) [noise=true , address=45.91.116.6:4975     , piece_id=FRO3TAUM45ES74OBKYQ542KQ5BIQWFOF62KJZZ5GQKG4INYCPNCA, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.015600875 (587h51m39.88668391s) [noise=true , address=145.79.227.43:4157   , piece_id=XRZDMN2TVETKMXIAP7XUQZZNXP22BECFSNSGSW4ZWM4K6JHXHX6A, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.015723789 (587h51m39.886806824s) [noise=true , address=145.79.227.52:4445   , piece_id=VUVWUFH3XLDTPTDZDBORNEISWN6YL524QWN4LKOVDEX5ESIM6UGQ, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.016010756 (587h51m39.887093801s) [noise=true , address=77.90.31.14:4661     , piece_id=LW6QJENC2ZNCSQ3AFHGN5ZOKMRCLY5EWHLATHJJRDFMLC3P45SUA, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.016271542 (587h51m39.887354587s) [noise=true , address=45.91.116.14:4191    , piece_id=SIGRDBV7INJ2VB4RBHVENZBZAGSHUHSBWV23ABQET3ILLHMVOSHQ, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.016548069 (587h51m39.887631114s) [noise=true , address=145.79.227.45:4230   , piece_id=UPEPHXZBSNL5O5V2OJSHKLIGTTCSQFPE2LGRYQ56NNIK3MP6XYRQ, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.016802413 (587h51m39.887885458s) [noise=true , address=145.79.227.20:4438   , piece_id=SUL7MIXOHA5XSTCB35INWSGRSYLE4YRTEZG27VJJ5BXDOP3YNC7Q, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.017067007 (587h51m39.888150042s) [noise=true , address=45.91.116.8:4048     , piece_id=UYY22G2SOK4TU2YJM5GQKK233GTUX3CLI7I5X74G6C2PX72FLNGA, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.017327422 (587h51m39.888410467s) [noise=true , address=45.91.116.5:4297     , piece_id=ZDS2FR3OVVVYC5SI4H4QABVNW7QHRFP6DBOXRFQK5DLP2DQCPZEA, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.018480409 (587h51m39.889563454s) [noise=true , address=145.79.227.39:4014   , piece_id=ZBFQGON265Q73ZECLEEAOTV4RBFW7WMNGM4IHLYWTZWMMC7D6KYA, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.018891561 (587h51m39.889974606s) [noise=true , address=145.79.227.41:4085   , piece_id=7AREJ7LQVLDLVVBMKD7VSZG4YNQTBTRCXMUJ5G3KHJQTEZ5O44IQ, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.019189719 (587h51m39.890272754s) [noise=true , address=45.91.116.10:4081    , piece_id=3K7P6VSFLLHDK5SSVL7CAW266M35ZIPJMKM47SDTZDQNZKLWQ3FA, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.020041601 (587h51m39.891124646s) [noise=true , address=145.79.227.9:4365    , piece_id=7BBEWSTPDOBBIQU6AKW5732WJ2WNNKG3QF5APWHBIAIGRXAWH5TQ, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.020239878 (587h51m39.891322923s) [noise=true , address=145.79.227.10:4402   , piece_id=VQRYJEZPBMKE3UVQLRYVWONRDNGWINVEQDVTEOF44IC3LGUQPPNQ, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:40.020662102 (587h51m39.891745137s) [noise=true , address=145.79.227.50:4375   , piece_id=65F6LUAIIW23QNPJZFWJDSKUB5R3Y2YNXKYEXKDKLWKPLFCQNROQ, seg_pos={0 0}, upload=515329]: Uploading piece...
Dec 30 03:20:41.721195777 (587h51m41.592278812s) [upload=515329]: Next returned nil segment
Dec 30 03:20:41.721236074 (587h51m41.592319109s) [upload=515329]: Waiting for error group managing segments...
Dec 30 03:20:41.999999835 (587h51m41.87108287s) [noise=true , address=145.79.227.50:4375   , piece_id=65F6LUAIIW23QNPJZFWJDSKUB5R3Y2YNXKYEXKDKLWKPLFCQNROQ, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.012786243 (587h51m41.883869288s) [noise=true , address=145.79.227.10:4402   , piece_id=VQRYJEZPBMKE3UVQLRYVWONRDNGWINVEQDVTEOF44IC3LGUQPPNQ, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.017997135 (587h51m41.88908018s) [noise=true , address=145.79.227.52:4445   , piece_id=VUVWUFH3XLDTPTDZDBORNEISWN6YL524QWN4LKOVDEX5ESIM6UGQ, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.019923945 (587h51m41.89100699s) [noise=true , address=145.79.227.45:4230   , piece_id=UPEPHXZBSNL5O5V2OJSHKLIGTTCSQFPE2LGRYQ56NNIK3MP6XYRQ, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.030396306 (587h51m41.901479351s) [noise=true , address=145.79.227.41:4085   , piece_id=7AREJ7LQVLDLVVBMKD7VSZG4YNQTBTRCXMUJ5G3KHJQTEZ5O44IQ, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.033660902 (587h51m41.904743937s) [noise=true , address=145.79.227.20:4438   , piece_id=SUL7MIXOHA5XSTCB35INWSGRSYLE4YRTEZG27VJJ5BXDOP3YNC7Q, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.044122713 (587h51m41.915205748s) [noise=true , address=145.79.227.9:4365    , piece_id=7BBEWSTPDOBBIQU6AKW5732WJ2WNNKG3QF5APWHBIAIGRXAWH5TQ, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.049098697 (587h51m41.920181742s) [noise=true , address=145.79.227.39:4014   , piece_id=ZBFQGON265Q73ZECLEEAOTV4RBFW7WMNGM4IHLYWTZWMMC7D6KYA, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.267014175 (587h51m42.13809722s) [noise=true , address=45.91.116.5:4297     , piece_id=ZDS2FR3OVVVYC5SI4H4QABVNW7QHRFP6DBOXRFQK5DLP2DQCPZEA, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.347944229 (587h51m42.219027274s) [noise=true , address=45.91.116.14:4191    , piece_id=SIGRDBV7INJ2VB4RBHVENZBZAGSHUHSBWV23ABQET3ILLHMVOSHQ, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.369330091 (587h51m42.240413126s) [noise=true , address=45.91.116.6:4975     , piece_id=FRO3TAUM45ES74OBKYQ542KQ5BIQWFOF62KJZZ5GQKG4INYCPNCA, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.399361817 (587h51m42.270444862s) [noise=true , address=45.91.116.8:4048     , piece_id=UYY22G2SOK4TU2YJM5GQKK233GTUX3CLI7I5X74G6C2PX72FLNGA, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.419032833 (587h51m42.290115868s) [noise=true , address=45.91.116.10:4081    , piece_id=3K7P6VSFLLHDK5SSVL7CAW266M35ZIPJMKM47SDTZDQNZKLWQ3FA, seg_pos={0 0}, upload=515329]: Done uploading piece. err: <nil>
Dec 30 03:20:42.419056548 (587h51m42.290139593s) [seg_pos={0 0}, upload=515329]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:20:42.419288260 (587h51m42.290371305s) [noise=true , address=145.79.227.43:4157   , piece_id=XRZDMN2TVETKMXIAP7XUQZZNXP22BECFSNSGSW4ZWM4K6JHXHX6A, seg_pos={0 0}, upload=515329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Dec 30 03:20:42.419296145 (587h51m42.29037919s) [noise=true , address=145.79.227.43:4157   , piece_id=XRZDMN2TVETKMXIAP7XUQZZNXP22BECFSNSGSW4ZWM4K6JHXHX6A, seg_pos={0 0}, upload=515329]: Long tail cancellation due to optimal threshold
Dec 30 03:20:42.419642865 (587h51m42.29072591s) [noise=true , address=77.90.31.19:4844     , piece_id=EKAGFL7AEGC3AMPDGQBAFWTC7NOD5N7CGWCNBJGTC5PQC4T34GFA, seg_pos={0 0}, upload=515329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:59452->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:59452->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59452->77.90.31.19:4844: use of closed network connection
Dec 30 03:20:42.419652242 (587h51m42.290735287s) [noise=true , address=77.90.31.19:4844     , piece_id=EKAGFL7AEGC3AMPDGQBAFWTC7NOD5N7CGWCNBJGTC5PQC4T34GFA, seg_pos={0 0}, upload=515329]: Long tail cancellation due to optimal threshold
Dec 30 03:20:42.419677902 (587h51m42.290760947s) [noise=true , address=77.90.31.14:4661     , piece_id=LW6QJENC2ZNCSQ3AFHGN5ZOKMRCLY5EWHLATHJJRDFMLC3P45SUA, seg_pos={0 0}, upload=515329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 30 03:20:42.419715844 (587h51m42.290798889s) [noise=true , address=77.90.31.14:4661     , piece_id=LW6QJENC2ZNCSQ3AFHGN5ZOKMRCLY5EWHLATHJJRDFMLC3P45SUA, seg_pos={0 0}, upload=515329]: Long tail cancellation due to optimal threshold
Dec 30 03:20:42.419749999 (587h51m42.290833034s) [seg_pos={0 0}, upload=515329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:20:42.419918069 (587h51m42.291001114s) [upload=515329]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:20:42.419928108 (587h51m42.291011153s) [upload=515329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:20:42.714936041 (587h51m42.586019076s) [upload=515329]: Done uploading segments...
Dec 30 03:20:42.714956601 (587h51m42.586039646s) [upload=515329]: Upload finished. err: <nil>
Duration: 30.002571159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...X................X......X................................]
duration: 
44.364901846s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:20:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 44.364928076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [...............X...X.X.XX................XX............X....]
duration: 
3.527639195s
trace: 
https://jaeger.dev.storj.io/trace/18e17e5647be4f14
FAIL (2025-12-30 03:20:40 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.527682737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X.X...]
duration: 
5.874648185s
trace: 
https://jaeger.dev.storj.io/trace/5e2e4fcf633beefc
FAIL (2025-12-30 03:20:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ) context deadline exceeded
======LOGS======
segments...
Dec 30 03:17:24.775486182 (587h48m24.646569227s) [upload=515278]: Got next segment. Inline: false
Dec 30 03:17:24.775521940 (587h48m24.646604985s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:17:24.775527320 (587h48m24.646610355s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:17:25.020197932 (587h48m24.891280967s) [seg_pos={0 0}, upload=515278]: Begin upload segment...
Dec 30 03:17:25.020244661 (587h48m24.891327696s) [seg_pos={0 0}, upload=515278]: Done begin upload segment.
Dec 30 03:17:25.020265711 (587h48m24.891348747s) [noise=true , address=45.91.116.14:4189    , piece_id=M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020308823 (587h48m24.891391858s) [noise=true , address=145.79.227.38:4979   , piece_id=F6KA4CLW7LEMOKBWSPUNWIYCH77S5PDGHLGUCROKQISZLGXLSHJQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020377805 (587h48m24.89146085s) [noise=true , address=145.79.227.17:4970   , piece_id=AO6IU3CRES5ZUTG64D6OQAZUHARJIW4UGQAW6KZDRI56IALZXGSA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020786163 (587h48m24.891869208s) [noise=true , address=145.79.227.26:4655   , piece_id=F7KE3BTRCQGCZNOAUDEXZYU4WWTY6RLWC5BX3RZDEGFIZV3YT3LA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021099760 (587h48m24.892182795s) [noise=true , address=45.91.116.6:4974     , piece_id=67UMQ23ULCNSFURJVPUBKVIFRI7AMTKGXQMG2Y777VLD7BYYSHGQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021409580 (587h48m24.892492615s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021727615 (587h48m24.89281065s) [noise=true , address=145.79.227.5:4222    , piece_id=LGGLVDGCUJKR2Q2VUIDWCLF52LI4CVROT6PMTEEV6QZXCO4GK2ZQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022063936 (587h48m24.893146971s) [noise=true , address=145.79.227.42:4123   , piece_id=LA36DI7XDMIBRUK6ROLL4QWJ4CWBIL7TVGEJTZCMBBHWDK4ER7DQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022488574 (587h48m24.893571609s) [noise=true , address=77.90.31.19:4842     , piece_id=IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022840124 (587h48m24.893923159s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.023179350 (587h48m24.894262385s) [noise=true , address=145.79.227.3:4150    , piece_id=F5NNJDWNZGSSD32PDRWJ52KJ5JAA6PBVMY5DA2ZBGRKNXQ3MF32Q, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.024388612 (587h48m24.895471647s) [noise=true , address=45.91.116.11:4117    , piece_id=EGUTUEQPRLCYZP2OORIDDTM6GZ4TUUGZFQKAQ367NT574UEHP65Q, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.024437756 (587h48m24.895520801s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.025611161 (587h48m24.896694196s) [noise=true , address=145.79.227.39:4013   , piece_id=ZCE6YD2VVEAEABR5GCS5GLID3QEQGQFHYNY32GTW2P2C7EDSNQHQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.025731579 (587h48m24.896814614s) [noise=true , address=145.79.227.10:4402   , piece_id=3KHEDODJTYPJXWFXWGAQ4SC5ZPJT3PD56ZMPEVUIJ3PIKNYNSRLA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.027346043 (587h48m24.898429078s) [noise=true , address=145.79.227.13:4803   , piece_id=NY7GZZNN7ASFUQYEKKRDKXFB2MD7QOCBFU2AOPTMP2FXSQVO2DXQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:26.719125921 (587h48m26.590208956s) [upload=515278]: Next returned nil segment
Dec 30 03:17:26.719141501 (587h48m26.590224536s) [upload=515278]: Waiting for error group managing segments...
Dec 30 03:17:26.998973246 (587h48m26.870056281s) [noise=true , address=145.79.227.3:4150    , piece_id=F5NNJDWNZGSSD32PDRWJ52KJ5JAA6PBVMY5DA2ZBGRKNXQ3MF32Q, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.011373198 (587h48m26.882456233s) [noise=true , address=145.79.227.10:4402   , piece_id=3KHEDODJTYPJXWFXWGAQ4SC5ZPJT3PD56ZMPEVUIJ3PIKNYNSRLA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.015690208 (587h48m26.886773243s) [noise=true , address=145.79.227.42:4123   , piece_id=LA36DI7XDMIBRUK6ROLL4QWJ4CWBIL7TVGEJTZCMBBHWDK4ER7DQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.020846897 (587h48m26.891929932s) [noise=true , address=145.79.227.39:4013   , piece_id=ZCE6YD2VVEAEABR5GCS5GLID3QEQGQFHYNY32GTW2P2C7EDSNQHQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.031087347 (587h48m26.902170382s) [noise=true , address=145.79.227.26:4655   , piece_id=F7KE3BTRCQGCZNOAUDEXZYU4WWTY6RLWC5BX3RZDEGFIZV3YT3LA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.033859646 (587h48m26.904942681s) [noise=true , address=145.79.227.17:4970   , piece_id=AO6IU3CRES5ZUTG64D6OQAZUHARJIW4UGQAW6KZDRI56IALZXGSA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.035335235 (587h48m26.90641827s) [noise=true , address=145.79.227.38:4979   , piece_id=F6KA4CLW7LEMOKBWSPUNWIYCH77S5PDGHLGUCROKQISZLGXLSHJQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.040359483 (587h48m26.911442518s) [noise=true , address=145.79.227.13:4803   , piece_id=NY7GZZNN7ASFUQYEKKRDKXFB2MD7QOCBFU2AOPTMP2FXSQVO2DXQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.054520648 (587h48m26.925603693s) [noise=true , address=145.79.227.5:4222    , piece_id=LGGLVDGCUJKR2Q2VUIDWCLF52LI4CVROT6PMTEEV6QZXCO4GK2ZQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.276238736 (587h48m27.147321771s) [noise=true , address=45.91.116.6:4974     , piece_id=67UMQ23ULCNSFURJVPUBKVIFRI7AMTKGXQMG2Y777VLD7BYYSHGQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.280044552 (587h48m27.151127587s) [noise=true , address=45.91.116.14:4189    , piece_id=M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.318011517 (587h48m27.189094572s) [noise=true , address=77.90.31.19:4842     , piece_id=IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.365072900 (587h48m27.236155936s) [noise=true , address=45.91.116.11:4117    , piece_id=EGUTUEQPRLCYZP2OORIDDTM6GZ4TUUGZFQKAQ367NT574UEHP65Q, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.365091275 (587h48m27.23617432s) [seg_pos={0 0}, upload=515278]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:17:27.365268051 (587h48m27.236351097s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection
Dec 30 03:17:27.365273743 (587h48m27.236356778s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365455329 (587h48m27.236538374s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 30 03:17:27.365461821 (587h48m27.236544856s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365773154 (587h48m27.236856199s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection
Dec 30 03:17:27.365788492 (587h48m27.236871528s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365851833 (587h48m27.236934868s) [seg_pos={0 0}, upload=515278]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:17:27.365987983 (587h48m27.237071018s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:17:27.365993393 (587h48m27.237076438s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:17:27.666497662 (587h48m27.537580697s) [upload=515278]: Done uploading segments...
Dec 30 03:17:27.666517760 (587h48m27.537600795s) [upload=515278]: Upload finished. err: <nil>
Duration: 5.874669475s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [................X...X.X.XX................XX............X...]
duration: 
3.487031632s
trace: 
https://jaeger.dev.storj.io/trace/18e17e5647be4f14
FAIL (2025-12-30 03:19:55 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.487059977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X.X..]
duration: 
5.660106161s
trace: 
https://jaeger.dev.storj.io/trace/5e2e4fcf633beefc
FAIL (2025-12-30 03:19:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ) context deadline exceeded
======LOGS======
segments...
Dec 30 03:17:24.775486182 (587h48m24.646569227s) [upload=515278]: Got next segment. Inline: false
Dec 30 03:17:24.775521940 (587h48m24.646604985s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:17:24.775527320 (587h48m24.646610355s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:17:25.020197932 (587h48m24.891280967s) [seg_pos={0 0}, upload=515278]: Begin upload segment...
Dec 30 03:17:25.020244661 (587h48m24.891327696s) [seg_pos={0 0}, upload=515278]: Done begin upload segment.
Dec 30 03:17:25.020265711 (587h48m24.891348747s) [noise=true , address=45.91.116.14:4189    , piece_id=M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020308823 (587h48m24.891391858s) [noise=true , address=145.79.227.38:4979   , piece_id=F6KA4CLW7LEMOKBWSPUNWIYCH77S5PDGHLGUCROKQISZLGXLSHJQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020377805 (587h48m24.89146085s) [noise=true , address=145.79.227.17:4970   , piece_id=AO6IU3CRES5ZUTG64D6OQAZUHARJIW4UGQAW6KZDRI56IALZXGSA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020786163 (587h48m24.891869208s) [noise=true , address=145.79.227.26:4655   , piece_id=F7KE3BTRCQGCZNOAUDEXZYU4WWTY6RLWC5BX3RZDEGFIZV3YT3LA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021099760 (587h48m24.892182795s) [noise=true , address=45.91.116.6:4974     , piece_id=67UMQ23ULCNSFURJVPUBKVIFRI7AMTKGXQMG2Y777VLD7BYYSHGQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021409580 (587h48m24.892492615s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021727615 (587h48m24.89281065s) [noise=true , address=145.79.227.5:4222    , piece_id=LGGLVDGCUJKR2Q2VUIDWCLF52LI4CVROT6PMTEEV6QZXCO4GK2ZQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022063936 (587h48m24.893146971s) [noise=true , address=145.79.227.42:4123   , piece_id=LA36DI7XDMIBRUK6ROLL4QWJ4CWBIL7TVGEJTZCMBBHWDK4ER7DQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022488574 (587h48m24.893571609s) [noise=true , address=77.90.31.19:4842     , piece_id=IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022840124 (587h48m24.893923159s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.023179350 (587h48m24.894262385s) [noise=true , address=145.79.227.3:4150    , piece_id=F5NNJDWNZGSSD32PDRWJ52KJ5JAA6PBVMY5DA2ZBGRKNXQ3MF32Q, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.024388612 (587h48m24.895471647s) [noise=true , address=45.91.116.11:4117    , piece_id=EGUTUEQPRLCYZP2OORIDDTM6GZ4TUUGZFQKAQ367NT574UEHP65Q, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.024437756 (587h48m24.895520801s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.025611161 (587h48m24.896694196s) [noise=true , address=145.79.227.39:4013   , piece_id=ZCE6YD2VVEAEABR5GCS5GLID3QEQGQFHYNY32GTW2P2C7EDSNQHQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.025731579 (587h48m24.896814614s) [noise=true , address=145.79.227.10:4402   , piece_id=3KHEDODJTYPJXWFXWGAQ4SC5ZPJT3PD56ZMPEVUIJ3PIKNYNSRLA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.027346043 (587h48m24.898429078s) [noise=true , address=145.79.227.13:4803   , piece_id=NY7GZZNN7ASFUQYEKKRDKXFB2MD7QOCBFU2AOPTMP2FXSQVO2DXQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:26.719125921 (587h48m26.590208956s) [upload=515278]: Next returned nil segment
Dec 30 03:17:26.719141501 (587h48m26.590224536s) [upload=515278]: Waiting for error group managing segments...
Dec 30 03:17:26.998973246 (587h48m26.870056281s) [noise=true , address=145.79.227.3:4150    , piece_id=F5NNJDWNZGSSD32PDRWJ52KJ5JAA6PBVMY5DA2ZBGRKNXQ3MF32Q, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.011373198 (587h48m26.882456233s) [noise=true , address=145.79.227.10:4402   , piece_id=3KHEDODJTYPJXWFXWGAQ4SC5ZPJT3PD56ZMPEVUIJ3PIKNYNSRLA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.015690208 (587h48m26.886773243s) [noise=true , address=145.79.227.42:4123   , piece_id=LA36DI7XDMIBRUK6ROLL4QWJ4CWBIL7TVGEJTZCMBBHWDK4ER7DQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.020846897 (587h48m26.891929932s) [noise=true , address=145.79.227.39:4013   , piece_id=ZCE6YD2VVEAEABR5GCS5GLID3QEQGQFHYNY32GTW2P2C7EDSNQHQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.031087347 (587h48m26.902170382s) [noise=true , address=145.79.227.26:4655   , piece_id=F7KE3BTRCQGCZNOAUDEXZYU4WWTY6RLWC5BX3RZDEGFIZV3YT3LA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.033859646 (587h48m26.904942681s) [noise=true , address=145.79.227.17:4970   , piece_id=AO6IU3CRES5ZUTG64D6OQAZUHARJIW4UGQAW6KZDRI56IALZXGSA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.035335235 (587h48m26.90641827s) [noise=true , address=145.79.227.38:4979   , piece_id=F6KA4CLW7LEMOKBWSPUNWIYCH77S5PDGHLGUCROKQISZLGXLSHJQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.040359483 (587h48m26.911442518s) [noise=true , address=145.79.227.13:4803   , piece_id=NY7GZZNN7ASFUQYEKKRDKXFB2MD7QOCBFU2AOPTMP2FXSQVO2DXQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.054520648 (587h48m26.925603693s) [noise=true , address=145.79.227.5:4222    , piece_id=LGGLVDGCUJKR2Q2VUIDWCLF52LI4CVROT6PMTEEV6QZXCO4GK2ZQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.276238736 (587h48m27.147321771s) [noise=true , address=45.91.116.6:4974     , piece_id=67UMQ23ULCNSFURJVPUBKVIFRI7AMTKGXQMG2Y777VLD7BYYSHGQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.280044552 (587h48m27.151127587s) [noise=true , address=45.91.116.14:4189    , piece_id=M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.318011517 (587h48m27.189094572s) [noise=true , address=77.90.31.19:4842     , piece_id=IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.365072900 (587h48m27.236155936s) [noise=true , address=45.91.116.11:4117    , piece_id=EGUTUEQPRLCYZP2OORIDDTM6GZ4TUUGZFQKAQ367NT574UEHP65Q, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.365091275 (587h48m27.23617432s) [seg_pos={0 0}, upload=515278]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:17:27.365268051 (587h48m27.236351097s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection
Dec 30 03:17:27.365273743 (587h48m27.236356778s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365455329 (587h48m27.236538374s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 30 03:17:27.365461821 (587h48m27.236544856s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365773154 (587h48m27.236856199s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection
Dec 30 03:17:27.365788492 (587h48m27.236871528s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365851833 (587h48m27.236934868s) [seg_pos={0 0}, upload=515278]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:17:27.365987983 (587h48m27.237071018s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:17:27.365993393 (587h48m27.237076438s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:17:27.666497662 (587h48m27.537580697s) [upload=515278]: Done uploading segments...
Dec 30 03:17:27.666517760 (587h48m27.537600795s) [upload=515278]: Upload finished. err: <nil>
Duration: 5.660135957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [....X................X......X...............................]
duration: 
25.977663232s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:19:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 25.977692197s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [.................X...X.X.XX................XX............X..]
duration: 
3.369920504s
trace: 
https://jaeger.dev.storj.io/trace/18e17e5647be4f14
FAIL (2025-12-30 03:19:15 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.369939779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [..................X...X.X.XX................XX............X.]
duration: 
3.790032212s
trace: 
https://jaeger.dev.storj.io/trace/18e17e5647be4f14
FAIL (2025-12-30 03:18:53 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.790058985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [...................X...X.X.XX................XX............X]
duration: 
5.001658498s
trace: 
https://jaeger.dev.storj.io/trace/18e17e5647be4f14
FAIL (2025-12-30 03:18:39 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001683166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.....X................X......X..............................]
duration: 
22.297416067s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:18:24 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 22.297689829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [.X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X.X.]
duration: 
5.941185361s
trace: 
https://jaeger.dev.storj.io/trace/5e2e4fcf633beefc
FAIL (2025-12-30 03:18:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ) context deadline exceeded
======LOGS======
segments...
Dec 30 03:17:24.775486182 (587h48m24.646569227s) [upload=515278]: Got next segment. Inline: false
Dec 30 03:17:24.775521940 (587h48m24.646604985s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:17:24.775527320 (587h48m24.646610355s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:17:25.020197932 (587h48m24.891280967s) [seg_pos={0 0}, upload=515278]: Begin upload segment...
Dec 30 03:17:25.020244661 (587h48m24.891327696s) [seg_pos={0 0}, upload=515278]: Done begin upload segment.
Dec 30 03:17:25.020265711 (587h48m24.891348747s) [noise=true , address=45.91.116.14:4189    , piece_id=M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020308823 (587h48m24.891391858s) [noise=true , address=145.79.227.38:4979   , piece_id=F6KA4CLW7LEMOKBWSPUNWIYCH77S5PDGHLGUCROKQISZLGXLSHJQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020377805 (587h48m24.89146085s) [noise=true , address=145.79.227.17:4970   , piece_id=AO6IU3CRES5ZUTG64D6OQAZUHARJIW4UGQAW6KZDRI56IALZXGSA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020786163 (587h48m24.891869208s) [noise=true , address=145.79.227.26:4655   , piece_id=F7KE3BTRCQGCZNOAUDEXZYU4WWTY6RLWC5BX3RZDEGFIZV3YT3LA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021099760 (587h48m24.892182795s) [noise=true , address=45.91.116.6:4974     , piece_id=67UMQ23ULCNSFURJVPUBKVIFRI7AMTKGXQMG2Y777VLD7BYYSHGQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021409580 (587h48m24.892492615s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021727615 (587h48m24.89281065s) [noise=true , address=145.79.227.5:4222    , piece_id=LGGLVDGCUJKR2Q2VUIDWCLF52LI4CVROT6PMTEEV6QZXCO4GK2ZQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022063936 (587h48m24.893146971s) [noise=true , address=145.79.227.42:4123   , piece_id=LA36DI7XDMIBRUK6ROLL4QWJ4CWBIL7TVGEJTZCMBBHWDK4ER7DQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022488574 (587h48m24.893571609s) [noise=true , address=77.90.31.19:4842     , piece_id=IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022840124 (587h48m24.893923159s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.023179350 (587h48m24.894262385s) [noise=true , address=145.79.227.3:4150    , piece_id=F5NNJDWNZGSSD32PDRWJ52KJ5JAA6PBVMY5DA2ZBGRKNXQ3MF32Q, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.024388612 (587h48m24.895471647s) [noise=true , address=45.91.116.11:4117    , piece_id=EGUTUEQPRLCYZP2OORIDDTM6GZ4TUUGZFQKAQ367NT574UEHP65Q, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.024437756 (587h48m24.895520801s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.025611161 (587h48m24.896694196s) [noise=true , address=145.79.227.39:4013   , piece_id=ZCE6YD2VVEAEABR5GCS5GLID3QEQGQFHYNY32GTW2P2C7EDSNQHQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.025731579 (587h48m24.896814614s) [noise=true , address=145.79.227.10:4402   , piece_id=3KHEDODJTYPJXWFXWGAQ4SC5ZPJT3PD56ZMPEVUIJ3PIKNYNSRLA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.027346043 (587h48m24.898429078s) [noise=true , address=145.79.227.13:4803   , piece_id=NY7GZZNN7ASFUQYEKKRDKXFB2MD7QOCBFU2AOPTMP2FXSQVO2DXQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:26.719125921 (587h48m26.590208956s) [upload=515278]: Next returned nil segment
Dec 30 03:17:26.719141501 (587h48m26.590224536s) [upload=515278]: Waiting for error group managing segments...
Dec 30 03:17:26.998973246 (587h48m26.870056281s) [noise=true , address=145.79.227.3:4150    , piece_id=F5NNJDWNZGSSD32PDRWJ52KJ5JAA6PBVMY5DA2ZBGRKNXQ3MF32Q, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.011373198 (587h48m26.882456233s) [noise=true , address=145.79.227.10:4402   , piece_id=3KHEDODJTYPJXWFXWGAQ4SC5ZPJT3PD56ZMPEVUIJ3PIKNYNSRLA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.015690208 (587h48m26.886773243s) [noise=true , address=145.79.227.42:4123   , piece_id=LA36DI7XDMIBRUK6ROLL4QWJ4CWBIL7TVGEJTZCMBBHWDK4ER7DQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.020846897 (587h48m26.891929932s) [noise=true , address=145.79.227.39:4013   , piece_id=ZCE6YD2VVEAEABR5GCS5GLID3QEQGQFHYNY32GTW2P2C7EDSNQHQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.031087347 (587h48m26.902170382s) [noise=true , address=145.79.227.26:4655   , piece_id=F7KE3BTRCQGCZNOAUDEXZYU4WWTY6RLWC5BX3RZDEGFIZV3YT3LA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.033859646 (587h48m26.904942681s) [noise=true , address=145.79.227.17:4970   , piece_id=AO6IU3CRES5ZUTG64D6OQAZUHARJIW4UGQAW6KZDRI56IALZXGSA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.035335235 (587h48m26.90641827s) [noise=true , address=145.79.227.38:4979   , piece_id=F6KA4CLW7LEMOKBWSPUNWIYCH77S5PDGHLGUCROKQISZLGXLSHJQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.040359483 (587h48m26.911442518s) [noise=true , address=145.79.227.13:4803   , piece_id=NY7GZZNN7ASFUQYEKKRDKXFB2MD7QOCBFU2AOPTMP2FXSQVO2DXQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.054520648 (587h48m26.925603693s) [noise=true , address=145.79.227.5:4222    , piece_id=LGGLVDGCUJKR2Q2VUIDWCLF52LI4CVROT6PMTEEV6QZXCO4GK2ZQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.276238736 (587h48m27.147321771s) [noise=true , address=45.91.116.6:4974     , piece_id=67UMQ23ULCNSFURJVPUBKVIFRI7AMTKGXQMG2Y777VLD7BYYSHGQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.280044552 (587h48m27.151127587s) [noise=true , address=45.91.116.14:4189    , piece_id=M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.318011517 (587h48m27.189094572s) [noise=true , address=77.90.31.19:4842     , piece_id=IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.365072900 (587h48m27.236155936s) [noise=true , address=45.91.116.11:4117    , piece_id=EGUTUEQPRLCYZP2OORIDDTM6GZ4TUUGZFQKAQ367NT574UEHP65Q, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.365091275 (587h48m27.23617432s) [seg_pos={0 0}, upload=515278]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:17:27.365268051 (587h48m27.236351097s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection
Dec 30 03:17:27.365273743 (587h48m27.236356778s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365455329 (587h48m27.236538374s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 30 03:17:27.365461821 (587h48m27.236544856s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365773154 (587h48m27.236856199s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection
Dec 30 03:17:27.365788492 (587h48m27.236871528s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365851833 (587h48m27.236934868s) [seg_pos={0 0}, upload=515278]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:17:27.365987983 (587h48m27.237071018s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:17:27.365993393 (587h48m27.237076438s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:17:27.666497662 (587h48m27.537580697s) [upload=515278]: Done uploading segments...
Dec 30 03:17:27.666517760 (587h48m27.537600795s) [upload=515278]: Upload finished. err: <nil>
Duration: 5.941209037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [..X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X.X]
duration: 
30.003755681s
trace: 
https://jaeger.dev.storj.io/trace/5e2e4fcf633beefc
FAIL (2025-12-30 03:17:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ) context deadline exceeded
======LOGS======
segments...
Dec 30 03:17:24.775486182 (587h48m24.646569227s) [upload=515278]: Got next segment. Inline: false
Dec 30 03:17:24.775521940 (587h48m24.646604985s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:17:24.775527320 (587h48m24.646610355s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:17:25.020197932 (587h48m24.891280967s) [seg_pos={0 0}, upload=515278]: Begin upload segment...
Dec 30 03:17:25.020244661 (587h48m24.891327696s) [seg_pos={0 0}, upload=515278]: Done begin upload segment.
Dec 30 03:17:25.020265711 (587h48m24.891348747s) [noise=true , address=45.91.116.14:4189    , piece_id=M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020308823 (587h48m24.891391858s) [noise=true , address=145.79.227.38:4979   , piece_id=F6KA4CLW7LEMOKBWSPUNWIYCH77S5PDGHLGUCROKQISZLGXLSHJQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020377805 (587h48m24.89146085s) [noise=true , address=145.79.227.17:4970   , piece_id=AO6IU3CRES5ZUTG64D6OQAZUHARJIW4UGQAW6KZDRI56IALZXGSA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.020786163 (587h48m24.891869208s) [noise=true , address=145.79.227.26:4655   , piece_id=F7KE3BTRCQGCZNOAUDEXZYU4WWTY6RLWC5BX3RZDEGFIZV3YT3LA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021099760 (587h48m24.892182795s) [noise=true , address=45.91.116.6:4974     , piece_id=67UMQ23ULCNSFURJVPUBKVIFRI7AMTKGXQMG2Y777VLD7BYYSHGQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021409580 (587h48m24.892492615s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.021727615 (587h48m24.89281065s) [noise=true , address=145.79.227.5:4222    , piece_id=LGGLVDGCUJKR2Q2VUIDWCLF52LI4CVROT6PMTEEV6QZXCO4GK2ZQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022063936 (587h48m24.893146971s) [noise=true , address=145.79.227.42:4123   , piece_id=LA36DI7XDMIBRUK6ROLL4QWJ4CWBIL7TVGEJTZCMBBHWDK4ER7DQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022488574 (587h48m24.893571609s) [noise=true , address=77.90.31.19:4842     , piece_id=IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.022840124 (587h48m24.893923159s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.023179350 (587h48m24.894262385s) [noise=true , address=145.79.227.3:4150    , piece_id=F5NNJDWNZGSSD32PDRWJ52KJ5JAA6PBVMY5DA2ZBGRKNXQ3MF32Q, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.024388612 (587h48m24.895471647s) [noise=true , address=45.91.116.11:4117    , piece_id=EGUTUEQPRLCYZP2OORIDDTM6GZ4TUUGZFQKAQ367NT574UEHP65Q, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.024437756 (587h48m24.895520801s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.025611161 (587h48m24.896694196s) [noise=true , address=145.79.227.39:4013   , piece_id=ZCE6YD2VVEAEABR5GCS5GLID3QEQGQFHYNY32GTW2P2C7EDSNQHQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.025731579 (587h48m24.896814614s) [noise=true , address=145.79.227.10:4402   , piece_id=3KHEDODJTYPJXWFXWGAQ4SC5ZPJT3PD56ZMPEVUIJ3PIKNYNSRLA, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:25.027346043 (587h48m24.898429078s) [noise=true , address=145.79.227.13:4803   , piece_id=NY7GZZNN7ASFUQYEKKRDKXFB2MD7QOCBFU2AOPTMP2FXSQVO2DXQ, seg_pos={0 0}, upload=515278]: Uploading piece...
Dec 30 03:17:26.719125921 (587h48m26.590208956s) [upload=515278]: Next returned nil segment
Dec 30 03:17:26.719141501 (587h48m26.590224536s) [upload=515278]: Waiting for error group managing segments...
Dec 30 03:17:26.998973246 (587h48m26.870056281s) [noise=true , address=145.79.227.3:4150    , piece_id=F5NNJDWNZGSSD32PDRWJ52KJ5JAA6PBVMY5DA2ZBGRKNXQ3MF32Q, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.011373198 (587h48m26.882456233s) [noise=true , address=145.79.227.10:4402   , piece_id=3KHEDODJTYPJXWFXWGAQ4SC5ZPJT3PD56ZMPEVUIJ3PIKNYNSRLA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.015690208 (587h48m26.886773243s) [noise=true , address=145.79.227.42:4123   , piece_id=LA36DI7XDMIBRUK6ROLL4QWJ4CWBIL7TVGEJTZCMBBHWDK4ER7DQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.020846897 (587h48m26.891929932s) [noise=true , address=145.79.227.39:4013   , piece_id=ZCE6YD2VVEAEABR5GCS5GLID3QEQGQFHYNY32GTW2P2C7EDSNQHQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.031087347 (587h48m26.902170382s) [noise=true , address=145.79.227.26:4655   , piece_id=F7KE3BTRCQGCZNOAUDEXZYU4WWTY6RLWC5BX3RZDEGFIZV3YT3LA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.033859646 (587h48m26.904942681s) [noise=true , address=145.79.227.17:4970   , piece_id=AO6IU3CRES5ZUTG64D6OQAZUHARJIW4UGQAW6KZDRI56IALZXGSA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.035335235 (587h48m26.90641827s) [noise=true , address=145.79.227.38:4979   , piece_id=F6KA4CLW7LEMOKBWSPUNWIYCH77S5PDGHLGUCROKQISZLGXLSHJQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.040359483 (587h48m26.911442518s) [noise=true , address=145.79.227.13:4803   , piece_id=NY7GZZNN7ASFUQYEKKRDKXFB2MD7QOCBFU2AOPTMP2FXSQVO2DXQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.054520648 (587h48m26.925603693s) [noise=true , address=145.79.227.5:4222    , piece_id=LGGLVDGCUJKR2Q2VUIDWCLF52LI4CVROT6PMTEEV6QZXCO4GK2ZQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.276238736 (587h48m27.147321771s) [noise=true , address=45.91.116.6:4974     , piece_id=67UMQ23ULCNSFURJVPUBKVIFRI7AMTKGXQMG2Y777VLD7BYYSHGQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.280044552 (587h48m27.151127587s) [noise=true , address=45.91.116.14:4189    , piece_id=M66PJVSE7Z4F4SJHQX66DEPUVFQHYT5AIQQXLDQG5OXR5YDZOFUA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.318011517 (587h48m27.189094572s) [noise=true , address=77.90.31.19:4842     , piece_id=IP2SJEZJ4BO5HO4FF4L336Z4TZP5Q7OQUYD2DYQKWAIUED5IHCGQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.365072900 (587h48m27.236155936s) [noise=true , address=45.91.116.11:4117    , piece_id=EGUTUEQPRLCYZP2OORIDDTM6GZ4TUUGZFQKAQ367NT574UEHP65Q, seg_pos={0 0}, upload=515278]: Done uploading piece. err: <nil>
Dec 30 03:17:27.365091275 (587h48m27.23617432s) [seg_pos={0 0}, upload=515278]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:17:27.365268051 (587h48m27.236351097s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33864->45.91.116.5:4299: use of closed network connection
Dec 30 03:17:27.365273743 (587h48m27.236356778s) [noise=true , address=45.91.116.5:4299     , piece_id=RW2QNNLAMBPDBK72GFKCEQOMYK6ZIQLZCJMT6ZQEFN74F66525UQ, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365455329 (587h48m27.236538374s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 30 03:17:27.365461821 (587h48m27.236544856s) [noise=true , address=45.91.116.13:4155    , piece_id=PDVDKALPGZGAU2QWF33KO2ONI26KWTPE2267UQSIVRX4FUHPXEXQ, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365773154 (587h48m27.236856199s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37066->77.90.31.13:4627: use of closed network connection
Dec 30 03:17:27.365788492 (587h48m27.236871528s) [noise=true , address=77.90.31.13:4627     , piece_id=ZWYZBQZP4D2QFNULKRUED62QH3UXGTYZFWCZPLLXIYATPUSBBTDA, seg_pos={0 0}, upload=515278]: Long tail cancellation due to optimal threshold
Dec 30 03:17:27.365851833 (587h48m27.236934868s) [seg_pos={0 0}, upload=515278]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:17:27.365987983 (587h48m27.237071018s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:17:27.365993393 (587h48m27.237076438s) [upload=515278]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:17:27.666497662 (587h48m27.537580697s) [upload=515278]: Done uploading segments...
Dec 30 03:17:27.666517760 (587h48m27.537600795s) [upload=515278]: Upload finished. err: <nil>
Duration: 30.003795426s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [....................X...X.X.XX................XX............]
duration: 
3.54267817s
trace: 
https://jaeger.dev.storj.io/trace/55b84c463909ac27
FAIL (2025-12-30 03:17:36 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.542708507s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [......X................X......X.............................]
duration: 
15.193026519s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:17:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 15.19304795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X.]
duration: 
6.086196998s
trace: 
https://jaeger.dev.storj.io/trace/5e9d1d8cfe4f4a4c
FAIL (2025-12-30 03:16:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 03:16:10.141517424 (587h47m10.012600469s) [upload=515262]: Starting upload
Dec 30 03:16:10.141594080 (587h47m10.012677125s) [upload=515262]: Done starting upload
Dec 30 03:16:10.141708167 (587h47m10.012791212s) [upload=515262]: Uploading segments...
Dec 30 03:16:10.141845849 (587h47m10.012928884s) [upload=515262]: Got next segment. Inline: false
Dec 30 03:16:10.141885905 (587h47m10.01296894s) [upload=515262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:16:10.141890986 (587h47m10.012974031s) [upload=515262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:16:10.396600755 (587h47m10.26768381s) [seg_pos={0 0}, upload=515262]: Begin upload segment...
Dec 30 03:16:10.396670057 (587h47m10.267753092s) [seg_pos={0 0}, upload=515262]: Done begin upload segment.
Dec 30 03:16:10.396700335 (587h47m10.26778337s) [noise=true , address=145.79.227.41:4085   , piece_id=GNMOHJ4XKQP5ARHR4O6UM2JDMG7RJRDXALS6Y33PIHJFEKP6PDUA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.396771220 (587h47m10.267854265s) [noise=true , address=45.91.116.9:4263     , piece_id=FZZ5JGV5NTKPJ7J7T3N6EISZILKJL7VIFMORI3JI7FOO3AGTRNUQ, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.397290569 (587h47m10.268373614s) [noise=true , address=145.79.227.47:4267   , piece_id=VXDGERY6WGW7PBVBUXKFIUWCBXFY3VGKASWIHEJC5HWBE5RRG33A, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.397707453 (587h47m10.268790498s) [noise=true , address=145.79.227.51:4409   , piece_id=CPZTKG46BFWNRCKMMLTGYV66J2K72PY2GP2Y54WRBKKSRC3V46LA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.398202154 (587h47m10.26928519s) [noise=true , address=145.79.227.10:4401   , piece_id=ZVTC4EMASDGXU2FKQGA6GEUNIV3JLH2FX6Q3WGS5LTZOR5I5VL5Q, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.398612877 (587h47m10.269695922s) [noise=true , address=77.90.31.11:4556     , piece_id=DVFEM24K5YAEOPK65EJZG6VL6CSUFKZBPYZNFIUANZMUBNJLUFFA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.399063966 (587h47m10.270147011s) [noise=true , address=145.79.227.17:4971   , piece_id=XDJ76QT7R74KU2J2X4T7WGMRM5W7DUN4TBFQCBV7UM565FKJ5VFQ, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.399080727 (587h47m10.270163762s) [noise=true , address=77.90.31.10:4518     , piece_id=S6HEUIFT7SHO3K5SONUS6QXKVVJBRW46HPRYBJGFYZHLDMRJA2HA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.399431315 (587h47m10.27051435s) [noise=true , address=145.79.227.7:4294    , piece_id=VYNTQRYEYLH3MBIL6BYO2G2TN4BR53ECZIG5ZTSAAWGWM36ET3XQ, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.399870702 (587h47m10.270953737s) [noise=true , address=145.79.227.36:4517   , piece_id=4DQFTW4V4YFAYZQTKGWYUION75ZSSEXSRLNHGXLGT462R4SA42UQ, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.400266896 (587h47m10.271349931s) [noise=true , address=45.91.116.14:4190    , piece_id=I2EUZCECQPGB6PCKI2QEJ45L22VH2HKCJH52TXZOKO76DUMUHROA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.400664163 (587h47m10.271747208s) [noise=true , address=45.91.116.6:4976     , piece_id=2N54F7MMJ2BXHYS23TICR465BE6O75KLXYLFJZSZOM5K2ERG3AYA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.401062591 (587h47m10.272145626s) [noise=true , address=45.91.116.5:4300     , piece_id=OHV72GS3O6DQVV2GSHGRAMTGPFFUD5OH2IBQX5MHXMQNRYPN2GNA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.401500325 (587h47m10.27258336s) [noise=true , address=145.79.227.3:4149    , piece_id=XAYF6HN4PUWGTKE7D4RY65TENNFBFQQPVXIE34WK4RJ5EJOHW3LA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.407099336 (587h47m10.278182371s) [noise=true , address=77.90.31.19:4841     , piece_id=BG42A2RJJEE5ZOOX45X7ZKJO3XL5QSYZDCZNF6FNCQUTHFTYNWDA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.408678803 (587h47m10.279761838s) [noise=true , address=145.79.227.50:4374   , piece_id=K22YI6TGYIW4QYDLZ5W4AZ2OUFPZPBI5ONYJQLT7LEPXZDMR33XQ, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:12.109361142 (587h47m11.980444187s) [upload=515262]: Next returned nil segment
Dec 30 03:16:12.109386350 (587h47m11.980469385s) [upload=515262]: Waiting for error group managing segments...
Dec 30 03:16:12.391023243 (587h47m12.262106278s) [noise=true , address=145.79.227.3:4149    , piece_id=XAYF6HN4PUWGTKE7D4RY65TENNFBFQQPVXIE34WK4RJ5EJOHW3LA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.391563602 (587h47m12.262646637s) [noise=true , address=145.79.227.50:4374   , piece_id=K22YI6TGYIW4QYDLZ5W4AZ2OUFPZPBI5ONYJQLT7LEPXZDMR33XQ, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.394937557 (587h47m12.266020602s) [noise=true , address=145.79.227.7:4294    , piece_id=VYNTQRYEYLH3MBIL6BYO2G2TN4BR53ECZIG5ZTSAAWGWM36ET3XQ, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.397717419 (587h47m12.268800464s) [noise=true , address=145.79.227.36:4517   , piece_id=4DQFTW4V4YFAYZQTKGWYUION75ZSSEXSRLNHGXLGT462R4SA42UQ, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.401982211 (587h47m12.273065246s) [noise=true , address=145.79.227.47:4267   , piece_id=VXDGERY6WGW7PBVBUXKFIUWCBXFY3VGKASWIHEJC5HWBE5RRG33A, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.405112141 (587h47m12.276195176s) [noise=true , address=145.79.227.10:4401   , piece_id=ZVTC4EMASDGXU2FKQGA6GEUNIV3JLH2FX6Q3WGS5LTZOR5I5VL5Q, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.408856179 (587h47m12.279939214s) [noise=true , address=145.79.227.41:4085   , piece_id=GNMOHJ4XKQP5ARHR4O6UM2JDMG7RJRDXALS6Y33PIHJFEKP6PDUA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.471813991 (587h47m12.342897026s) [noise=true , address=145.79.227.51:4409   , piece_id=CPZTKG46BFWNRCKMMLTGYV66J2K72PY2GP2Y54WRBKKSRC3V46LA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.477520758 (587h47m12.348603793s) [noise=true , address=145.79.227.17:4971   , piece_id=XDJ76QT7R74KU2J2X4T7WGMRM5W7DUN4TBFQCBV7UM565FKJ5VFQ, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.586835702 (587h47m12.457918747s) [noise=true , address=77.90.31.19:4841     , piece_id=BG42A2RJJEE5ZOOX45X7ZKJO3XL5QSYZDCZNF6FNCQUTHFTYNWDA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.595883992 (587h47m12.466967027s) [noise=true , address=77.90.31.10:4518     , piece_id=S6HEUIFT7SHO3K5SONUS6QXKVVJBRW46HPRYBJGFYZHLDMRJA2HA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.640102600 (587h47m12.511185646s) [noise=true , address=45.91.116.6:4976     , piece_id=2N54F7MMJ2BXHYS23TICR465BE6O75KLXYLFJZSZOM5K2ERG3AYA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.731706359 (587h47m12.602789404s) [noise=true , address=45.91.116.9:4263     , piece_id=FZZ5JGV5NTKPJ7J7T3N6EISZILKJL7VIFMORI3JI7FOO3AGTRNUQ, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.731731037 (587h47m12.602814072s) [seg_pos={0 0}, upload=515262]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:16:12.731952338 (587h47m12.603035383s) [noise=true , address=45.91.116.14:4190    , piece_id=I2EUZCECQPGB6PCKI2QEJ45L22VH2HKCJH52TXZOKO76DUMUHROA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF
Dec 30 03:16:12.731958509 (587h47m12.603041554s) [noise=true , address=45.91.116.14:4190    , piece_id=I2EUZCECQPGB6PCKI2QEJ45L22VH2HKCJH52TXZOKO76DUMUHROA, seg_pos={0 0}, upload=515262]: Long tail cancellation due to optimal threshold
Dec 30 03:16:12.731991692 (587h47m12.603074738s) [noise=true , address=77.90.31.11:4556     , piece_id=DVFEM24K5YAEOPK65EJZG6VL6CSUFKZBPYZNFIUANZMUBNJLUFFA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 30 03:16:12.732001322 (587h47m12.603084357s) [noise=true , address=77.90.31.11:4556     , piece_id=DVFEM24K5YAEOPK65EJZG6VL6CSUFKZBPYZNFIUANZMUBNJLUFFA, seg_pos={0 0}, upload=515262]: Long tail cancellation due to optimal threshold
Dec 30 03:16:12.732271395 (587h47m12.60335443s) [noise=true , address=45.91.116.5:4300     , piece_id=OHV72GS3O6DQVV2GSHGRAMTGPFFUD5OH2IBQX5MHXMQNRYPN2GNA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: read tcp 188.166.225.170:46652->45.91.116.5:4300: use of closed network connection; context canceled
Dec 30 03:16:12.732278459 (587h47m12.603361494s) [noise=true , address=45.91.116.5:4300     , piece_id=OHV72GS3O6DQVV2GSHGRAMTGPFFUD5OH2IBQX5MHXMQNRYPN2GNA, seg_pos={0 0}, upload=515262]: Long tail cancellation due to optimal threshold
Dec 30 03:16:12.732296814 (587h47m12.603379859s) [seg_pos={0 0}, upload=515262]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:16:12.732376856 (587h47m12.603459891s) [upload=515262]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:16:12.732383179 (587h47m12.603466214s) [upload=515262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:16:13.030552021 (587h47m12.901635056s) [upload=515262]: Done uploading segments...
Dec 30 03:16:13.030573892 (587h47m12.901656937s) [upload=515262]: Upload finished. err: <nil>
Duration: 6.086225042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.X]
duration: 
30.002899467s
trace: 
https://jaeger.dev.storj.io/trace/5e9d1d8cfe4f4a4c
FAIL (2025-12-30 03:16:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 03:16:10.141517424 (587h47m10.012600469s) [upload=515262]: Starting upload
Dec 30 03:16:10.141594080 (587h47m10.012677125s) [upload=515262]: Done starting upload
Dec 30 03:16:10.141708167 (587h47m10.012791212s) [upload=515262]: Uploading segments...
Dec 30 03:16:10.141845849 (587h47m10.012928884s) [upload=515262]: Got next segment. Inline: false
Dec 30 03:16:10.141885905 (587h47m10.01296894s) [upload=515262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:16:10.141890986 (587h47m10.012974031s) [upload=515262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:16:10.396600755 (587h47m10.26768381s) [seg_pos={0 0}, upload=515262]: Begin upload segment...
Dec 30 03:16:10.396670057 (587h47m10.267753092s) [seg_pos={0 0}, upload=515262]: Done begin upload segment.
Dec 30 03:16:10.396700335 (587h47m10.26778337s) [noise=true , address=145.79.227.41:4085   , piece_id=GNMOHJ4XKQP5ARHR4O6UM2JDMG7RJRDXALS6Y33PIHJFEKP6PDUA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.396771220 (587h47m10.267854265s) [noise=true , address=45.91.116.9:4263     , piece_id=FZZ5JGV5NTKPJ7J7T3N6EISZILKJL7VIFMORI3JI7FOO3AGTRNUQ, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.397290569 (587h47m10.268373614s) [noise=true , address=145.79.227.47:4267   , piece_id=VXDGERY6WGW7PBVBUXKFIUWCBXFY3VGKASWIHEJC5HWBE5RRG33A, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.397707453 (587h47m10.268790498s) [noise=true , address=145.79.227.51:4409   , piece_id=CPZTKG46BFWNRCKMMLTGYV66J2K72PY2GP2Y54WRBKKSRC3V46LA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.398202154 (587h47m10.26928519s) [noise=true , address=145.79.227.10:4401   , piece_id=ZVTC4EMASDGXU2FKQGA6GEUNIV3JLH2FX6Q3WGS5LTZOR5I5VL5Q, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.398612877 (587h47m10.269695922s) [noise=true , address=77.90.31.11:4556     , piece_id=DVFEM24K5YAEOPK65EJZG6VL6CSUFKZBPYZNFIUANZMUBNJLUFFA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.399063966 (587h47m10.270147011s) [noise=true , address=145.79.227.17:4971   , piece_id=XDJ76QT7R74KU2J2X4T7WGMRM5W7DUN4TBFQCBV7UM565FKJ5VFQ, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.399080727 (587h47m10.270163762s) [noise=true , address=77.90.31.10:4518     , piece_id=S6HEUIFT7SHO3K5SONUS6QXKVVJBRW46HPRYBJGFYZHLDMRJA2HA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.399431315 (587h47m10.27051435s) [noise=true , address=145.79.227.7:4294    , piece_id=VYNTQRYEYLH3MBIL6BYO2G2TN4BR53ECZIG5ZTSAAWGWM36ET3XQ, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.399870702 (587h47m10.270953737s) [noise=true , address=145.79.227.36:4517   , piece_id=4DQFTW4V4YFAYZQTKGWYUION75ZSSEXSRLNHGXLGT462R4SA42UQ, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.400266896 (587h47m10.271349931s) [noise=true , address=45.91.116.14:4190    , piece_id=I2EUZCECQPGB6PCKI2QEJ45L22VH2HKCJH52TXZOKO76DUMUHROA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.400664163 (587h47m10.271747208s) [noise=true , address=45.91.116.6:4976     , piece_id=2N54F7MMJ2BXHYS23TICR465BE6O75KLXYLFJZSZOM5K2ERG3AYA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.401062591 (587h47m10.272145626s) [noise=true , address=45.91.116.5:4300     , piece_id=OHV72GS3O6DQVV2GSHGRAMTGPFFUD5OH2IBQX5MHXMQNRYPN2GNA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.401500325 (587h47m10.27258336s) [noise=true , address=145.79.227.3:4149    , piece_id=XAYF6HN4PUWGTKE7D4RY65TENNFBFQQPVXIE34WK4RJ5EJOHW3LA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.407099336 (587h47m10.278182371s) [noise=true , address=77.90.31.19:4841     , piece_id=BG42A2RJJEE5ZOOX45X7ZKJO3XL5QSYZDCZNF6FNCQUTHFTYNWDA, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:10.408678803 (587h47m10.279761838s) [noise=true , address=145.79.227.50:4374   , piece_id=K22YI6TGYIW4QYDLZ5W4AZ2OUFPZPBI5ONYJQLT7LEPXZDMR33XQ, seg_pos={0 0}, upload=515262]: Uploading piece...
Dec 30 03:16:12.109361142 (587h47m11.980444187s) [upload=515262]: Next returned nil segment
Dec 30 03:16:12.109386350 (587h47m11.980469385s) [upload=515262]: Waiting for error group managing segments...
Dec 30 03:16:12.391023243 (587h47m12.262106278s) [noise=true , address=145.79.227.3:4149    , piece_id=XAYF6HN4PUWGTKE7D4RY65TENNFBFQQPVXIE34WK4RJ5EJOHW3LA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.391563602 (587h47m12.262646637s) [noise=true , address=145.79.227.50:4374   , piece_id=K22YI6TGYIW4QYDLZ5W4AZ2OUFPZPBI5ONYJQLT7LEPXZDMR33XQ, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.394937557 (587h47m12.266020602s) [noise=true , address=145.79.227.7:4294    , piece_id=VYNTQRYEYLH3MBIL6BYO2G2TN4BR53ECZIG5ZTSAAWGWM36ET3XQ, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.397717419 (587h47m12.268800464s) [noise=true , address=145.79.227.36:4517   , piece_id=4DQFTW4V4YFAYZQTKGWYUION75ZSSEXSRLNHGXLGT462R4SA42UQ, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.401982211 (587h47m12.273065246s) [noise=true , address=145.79.227.47:4267   , piece_id=VXDGERY6WGW7PBVBUXKFIUWCBXFY3VGKASWIHEJC5HWBE5RRG33A, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.405112141 (587h47m12.276195176s) [noise=true , address=145.79.227.10:4401   , piece_id=ZVTC4EMASDGXU2FKQGA6GEUNIV3JLH2FX6Q3WGS5LTZOR5I5VL5Q, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.408856179 (587h47m12.279939214s) [noise=true , address=145.79.227.41:4085   , piece_id=GNMOHJ4XKQP5ARHR4O6UM2JDMG7RJRDXALS6Y33PIHJFEKP6PDUA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.471813991 (587h47m12.342897026s) [noise=true , address=145.79.227.51:4409   , piece_id=CPZTKG46BFWNRCKMMLTGYV66J2K72PY2GP2Y54WRBKKSRC3V46LA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.477520758 (587h47m12.348603793s) [noise=true , address=145.79.227.17:4971   , piece_id=XDJ76QT7R74KU2J2X4T7WGMRM5W7DUN4TBFQCBV7UM565FKJ5VFQ, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.586835702 (587h47m12.457918747s) [noise=true , address=77.90.31.19:4841     , piece_id=BG42A2RJJEE5ZOOX45X7ZKJO3XL5QSYZDCZNF6FNCQUTHFTYNWDA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.595883992 (587h47m12.466967027s) [noise=true , address=77.90.31.10:4518     , piece_id=S6HEUIFT7SHO3K5SONUS6QXKVVJBRW46HPRYBJGFYZHLDMRJA2HA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.640102600 (587h47m12.511185646s) [noise=true , address=45.91.116.6:4976     , piece_id=2N54F7MMJ2BXHYS23TICR465BE6O75KLXYLFJZSZOM5K2ERG3AYA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.731706359 (587h47m12.602789404s) [noise=true , address=45.91.116.9:4263     , piece_id=FZZ5JGV5NTKPJ7J7T3N6EISZILKJL7VIFMORI3JI7FOO3AGTRNUQ, seg_pos={0 0}, upload=515262]: Done uploading piece. err: <nil>
Dec 30 03:16:12.731731037 (587h47m12.602814072s) [seg_pos={0 0}, upload=515262]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:16:12.731952338 (587h47m12.603035383s) [noise=true , address=45.91.116.14:4190    , piece_id=I2EUZCECQPGB6PCKI2QEJ45L22VH2HKCJH52TXZOKO76DUMUHROA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF
Dec 30 03:16:12.731958509 (587h47m12.603041554s) [noise=true , address=45.91.116.14:4190    , piece_id=I2EUZCECQPGB6PCKI2QEJ45L22VH2HKCJH52TXZOKO76DUMUHROA, seg_pos={0 0}, upload=515262]: Long tail cancellation due to optimal threshold
Dec 30 03:16:12.731991692 (587h47m12.603074738s) [noise=true , address=77.90.31.11:4556     , piece_id=DVFEM24K5YAEOPK65EJZG6VL6CSUFKZBPYZNFIUANZMUBNJLUFFA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 30 03:16:12.732001322 (587h47m12.603084357s) [noise=true , address=77.90.31.11:4556     , piece_id=DVFEM24K5YAEOPK65EJZG6VL6CSUFKZBPYZNFIUANZMUBNJLUFFA, seg_pos={0 0}, upload=515262]: Long tail cancellation due to optimal threshold
Dec 30 03:16:12.732271395 (587h47m12.60335443s) [noise=true , address=45.91.116.5:4300     , piece_id=OHV72GS3O6DQVV2GSHGRAMTGPFFUD5OH2IBQX5MHXMQNRYPN2GNA, seg_pos={0 0}, upload=515262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: read tcp 188.166.225.170:46652->45.91.116.5:4300: use of closed network connection; context canceled
Dec 30 03:16:12.732278459 (587h47m12.603361494s) [noise=true , address=45.91.116.5:4300     , piece_id=OHV72GS3O6DQVV2GSHGRAMTGPFFUD5OH2IBQX5MHXMQNRYPN2GNA, seg_pos={0 0}, upload=515262]: Long tail cancellation due to optimal threshold
Dec 30 03:16:12.732296814 (587h47m12.603379859s) [seg_pos={0 0}, upload=515262]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:16:12.732376856 (587h47m12.603459891s) [upload=515262]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:16:12.732383179 (587h47m12.603466214s) [upload=515262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:16:13.030552021 (587h47m12.901635056s) [upload=515262]: Done uploading segments...
Dec 30 03:16:13.030573892 (587h47m12.901656937s) [upload=515262]: Upload finished. err: <nil>
Duration: 30.002925225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.......X................X......X............................]
duration: 
37.106073244s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:16:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 37.106096169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [.....................X...X.X.XX................XX...........]
duration: 
3.460608155s
trace: 
https://jaeger.dev.storj.io/trace/55b84c463909ac27
FAIL (2025-12-30 03:16:26 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.460647199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [........X................X......X...........................]
duration: 
34.634381719s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:15:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 34.634400856s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [......................X...X.X.XX................XX..........]
duration: 
3.437301846s
trace: 
https://jaeger.dev.storj.io/trace/55b84c463909ac27
FAIL (2025-12-30 03:15:39 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.437345659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.........X................X......X..........................]
duration: 
39.553230254s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:15:13 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 39.553275471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X.]
duration: 
6.310215445s
trace: 
https://jaeger.dev.storj.io/trace/767fc1eb960624d
FAIL (2025-12-30 03:15:02 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 5AS6ME2MO6NKTLYQ6NFAE72POU6XGGGOAVTHJXCBLWK75LFA7DPA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GZ3R7DIUPHK4AXJBYQ5C5PBQU2DL25OI5PT4QNIKW7BAPY4BNGEA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: L7K34LTYVM57KFEDJZEN7L2Y2HFAVQU5EDZXKIWS7ED5JFLDFZOQ) context deadline exceeded
======LOGS======
03:14:27.130309192 (587h45m27.001392237s) [upload=515239]: Starting upload
Dec 30 03:14:27.130389495 (587h45m27.00147254s) [upload=515239]: Done starting upload
Dec 30 03:14:27.130647727 (587h45m27.001730762s) [upload=515239]: Uploading segments...
Dec 30 03:14:27.130713231 (587h45m27.001796266s) [upload=515239]: Got next segment. Inline: false
Dec 30 03:14:27.130763437 (587h45m27.001846482s) [upload=515239]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:14:27.130769489 (587h45m27.001852534s) [upload=515239]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:14:27.376774924 (587h45m27.247857969s) [seg_pos={0 0}, upload=515239]: Begin upload segment...
Dec 30 03:14:27.376841731 (587h45m27.247924766s) [seg_pos={0 0}, upload=515239]: Done begin upload segment.
Dec 30 03:14:27.376896525 (587h45m27.24797957s) [noise=true , address=145.79.227.26:4654   , piece_id=6HME5K7VW5NCUPFCBRXDIHODOPECLPNWQJQ6I42OTQK3RBHLARIA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.376923176 (587h45m27.248006221s) [noise=true , address=145.79.227.46:4603   , piece_id=IEXLEOQE67PZTPA23JJIONBLH7XZGAQM5PBKQR6OGWZWVJGJYM7A, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.377735202 (587h45m27.248818247s) [noise=true , address=145.79.227.14:4837   , piece_id=23GZDADUIFN2OTV5BNPRMJ5MIKXJLRRDLOB5O5KDYOE3HJLGNIJA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.377896930 (587h45m27.248979975s) [noise=true , address=77.90.31.15:4700     , piece_id=L7K34LTYVM57KFEDJZEN7L2Y2HFAVQU5EDZXKIWS7ED5JFLDFZOQ, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.378408214 (587h45m27.249491259s) [noise=true , address=145.79.227.37:4565   , piece_id=YBOER4PYDEOSZZZIWMS6TYVIMFF34DMCQY45YHLSZISTM6AXOIFQ, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.378462918 (587h45m27.249545953s) [noise=true , address=45.91.116.11:4119    , piece_id=YEV2YPPCCLLMCHZY65ZMGU66U2JIPKCLIJCZCO3D45M7VSPLXYMA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.378821231 (587h45m27.249904276s) [noise=true , address=45.91.116.4:4939     , piece_id=FMI6IF3JBW6FOWU7VRQEFZCLSXTP6AY6ETVJLIEP5IQ3UPD5GNRQ, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.379250237 (587h45m27.250333282s) [noise=true , address=145.79.227.15:4876   , piece_id=5RMWWNADUHKU4EQCJ2ZZ5ZV45DN6BPGDPM7MLLEFDEVPHZYQYWQA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.379697038 (587h45m27.250780083s) [noise=true , address=45.91.116.5:4300     , piece_id=KU6Q4YAWNYHXQ4ERXL3W64ERC4B5OERR35BYPEIQ3FZEGPQIVXKA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.380171711 (587h45m27.251254746s) [noise=true , address=145.79.227.5:4221    , piece_id=N5TRSB7L23PL4N4KTRKFY7UEXEHQ5S77U6YVBLA4RK5MYVBQBN7Q, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.380555452 (587h45m27.251638497s) [noise=true , address=45.91.116.14:4190    , piece_id=62F4NHRQWURFCQRJSJ6O2LDKLRZVVNITWGE7IUEQDJBHGWSYF6YQ, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.381333154 (587h45m27.252416199s) [noise=true , address=45.91.116.8:4047     , piece_id=5AS6ME2MO6NKTLYQ6NFAE72POU6XGGGOAVTHJXCBLWK75LFA7DPA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.381746290 (587h45m27.252829335s) [noise=true , address=45.91.116.6:4973     , piece_id=GZ3R7DIUPHK4AXJBYQ5C5PBQU2DL25OI5PT4QNIKW7BAPY4BNGEA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.382331794 (587h45m27.25341484s) [noise=true , address=145.79.227.21:4475   , piece_id=ZFVBWT65I6QYW6U2CXSZZ4UWXZWLNAVHEJ5J6O5UR4UKPTJRZFAA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.382838620 (587h45m27.253921665s) [noise=true , address=77.90.31.14:4661     , piece_id=QXRYXOLIF4BRHJOPAWEMWRIZSYZAXGBKQCUHSZ3F3PWNRBKSTUMA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.383097222 (587h45m27.254180267s) [noise=true , address=77.90.31.16:4736     , piece_id=UBRN5RFF7UOODV74N7HBN6RKSHWA4LRFCWROHH6UNSXQ4UDXDXTQ, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:29.082585888 (587h45m28.953668923s) [upload=515239]: Next returned nil segment
Dec 30 03:14:29.082599774 (587h45m28.953682809s) [upload=515239]: Waiting for error group managing segments...
Dec 30 03:14:29.356719242 (587h45m29.227802277s) [noise=true , address=145.79.227.14:4837   , piece_id=23GZDADUIFN2OTV5BNPRMJ5MIKXJLRRDLOB5O5KDYOE3HJLGNIJA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.361320505 (587h45m29.23240354s) [noise=true , address=145.79.227.15:4876   , piece_id=5RMWWNADUHKU4EQCJ2ZZ5ZV45DN6BPGDPM7MLLEFDEVPHZYQYWQA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.371841238 (587h45m29.242924303s) [noise=true , address=145.79.227.21:4475   , piece_id=ZFVBWT65I6QYW6U2CXSZZ4UWXZWLNAVHEJ5J6O5UR4UKPTJRZFAA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.377008917 (587h45m29.248091952s) [noise=true , address=145.79.227.5:4221    , piece_id=N5TRSB7L23PL4N4KTRKFY7UEXEHQ5S77U6YVBLA4RK5MYVBQBN7Q, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.380165918 (587h45m29.251248983s) [noise=true , address=145.79.227.46:4603   , piece_id=IEXLEOQE67PZTPA23JJIONBLH7XZGAQM5PBKQR6OGWZWVJGJYM7A, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.389672331 (587h45m29.260755366s) [noise=true , address=145.79.227.26:4654   , piece_id=6HME5K7VW5NCUPFCBRXDIHODOPECLPNWQJQ6I42OTQK3RBHLARIA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.607462068 (587h45m29.478545113s) [noise=true , address=45.91.116.4:4939     , piece_id=FMI6IF3JBW6FOWU7VRQEFZCLSXTP6AY6ETVJLIEP5IQ3UPD5GNRQ, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.643311411 (587h45m29.514394446s) [noise=true , address=45.91.116.8:4047     , piece_id=5AS6ME2MO6NKTLYQ6NFAE72POU6XGGGOAVTHJXCBLWK75LFA7DPA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.657020886 (587h45m29.528103921s) [noise=true , address=77.90.31.14:4661     , piece_id=QXRYXOLIF4BRHJOPAWEMWRIZSYZAXGBKQCUHSZ3F3PWNRBKSTUMA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.674864493 (587h45m29.545947528s) [noise=true , address=45.91.116.6:4973     , piece_id=GZ3R7DIUPHK4AXJBYQ5C5PBQU2DL25OI5PT4QNIKW7BAPY4BNGEA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.851103982 (587h45m29.722187027s) [noise=true , address=45.91.116.11:4119    , piece_id=YEV2YPPCCLLMCHZY65ZMGU66U2JIPKCLIJCZCO3D45M7VSPLXYMA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.972339214 (587h45m29.843422269s) [noise=true , address=77.90.31.15:4700     , piece_id=L7K34LTYVM57KFEDJZEN7L2Y2HFAVQU5EDZXKIWS7ED5JFLDFZOQ, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:30.321145789 (587h45m30.192228824s) [noise=true , address=77.90.31.16:4736     , piece_id=UBRN5RFF7UOODV74N7HBN6RKSHWA4LRFCWROHH6UNSXQ4UDXDXTQ, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:30.321166298 (587h45m30.192249344s) [seg_pos={0 0}, upload=515239]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:14:30.321354677 (587h45m30.192437712s) [noise=true , address=45.91.116.14:4190    , piece_id=62F4NHRQWURFCQRJSJ6O2LDKLRZVVNITWGE7IUEQDJBHGWSYF6YQ, seg_pos={0 0}, upload=515239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF
Dec 30 03:14:30.321360648 (587h45m30.192443703s) [noise=true , address=45.91.116.14:4190    , piece_id=62F4NHRQWURFCQRJSJ6O2LDKLRZVVNITWGE7IUEQDJBHGWSYF6YQ, seg_pos={0 0}, upload=515239]: Long tail cancellation due to optimal threshold
Dec 30 03:14:30.321444969 (587h45m30.192528024s) [noise=true , address=145.79.227.37:4565   , piece_id=YBOER4PYDEOSZZZIWMS6TYVIMFF34DMCQY45YHLSZISTM6AXOIFQ, seg_pos={0 0}, upload=515239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:38988->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:38988->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->145.79.227.37:4565: use of closed network connection
Dec 30 03:14:30.321462823 (587h45m30.192545858s) [noise=true , address=145.79.227.37:4565   , piece_id=YBOER4PYDEOSZZZIWMS6TYVIMFF34DMCQY45YHLSZISTM6AXOIFQ, seg_pos={0 0}, upload=515239]: Long tail cancellation due to optimal threshold
Dec 30 03:14:30.321540551 (587h45m30.192623596s) [noise=true , address=45.91.116.5:4300     , piece_id=KU6Q4YAWNYHXQ4ERXL3W64ERC4B5OERR35BYPEIQ3FZEGPQIVXKA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 30 03:14:30.321547324 (587h45m30.192630369s) [noise=true , address=45.91.116.5:4300     , piece_id=KU6Q4YAWNYHXQ4ERXL3W64ERC4B5OERR35BYPEIQ3FZEGPQIVXKA, seg_pos={0 0}, upload=515239]: Long tail cancellation due to optimal threshold
Dec 30 03:14:30.321563635 (587h45m30.19264667s) [seg_pos={0 0}, upload=515239]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:14:30.321631394 (587h45m30.192714439s) [upload=515239]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:14:30.321635712 (587h45m30.192718747s) [upload=515239]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:14:30.608229163 (587h45m30.479312218s) [upload=515239]: Done uploading segments...
Dec 30 03:14:30.608256105 (587h45m30.47933915s) [upload=515239]: Upload finished. err: <nil>
Duration: 6.310233068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [..X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..X]
duration: 
30.002629737s
trace: 
https://jaeger.dev.storj.io/trace/767fc1eb960624d
FAIL (2025-12-30 03:14:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 5AS6ME2MO6NKTLYQ6NFAE72POU6XGGGOAVTHJXCBLWK75LFA7DPA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GZ3R7DIUPHK4AXJBYQ5C5PBQU2DL25OI5PT4QNIKW7BAPY4BNGEA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: L7K34LTYVM57KFEDJZEN7L2Y2HFAVQU5EDZXKIWS7ED5JFLDFZOQ) context deadline exceeded
======LOGS======
03:14:27.130309192 (587h45m27.001392237s) [upload=515239]: Starting upload
Dec 30 03:14:27.130389495 (587h45m27.00147254s) [upload=515239]: Done starting upload
Dec 30 03:14:27.130647727 (587h45m27.001730762s) [upload=515239]: Uploading segments...
Dec 30 03:14:27.130713231 (587h45m27.001796266s) [upload=515239]: Got next segment. Inline: false
Dec 30 03:14:27.130763437 (587h45m27.001846482s) [upload=515239]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:14:27.130769489 (587h45m27.001852534s) [upload=515239]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:14:27.376774924 (587h45m27.247857969s) [seg_pos={0 0}, upload=515239]: Begin upload segment...
Dec 30 03:14:27.376841731 (587h45m27.247924766s) [seg_pos={0 0}, upload=515239]: Done begin upload segment.
Dec 30 03:14:27.376896525 (587h45m27.24797957s) [noise=true , address=145.79.227.26:4654   , piece_id=6HME5K7VW5NCUPFCBRXDIHODOPECLPNWQJQ6I42OTQK3RBHLARIA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.376923176 (587h45m27.248006221s) [noise=true , address=145.79.227.46:4603   , piece_id=IEXLEOQE67PZTPA23JJIONBLH7XZGAQM5PBKQR6OGWZWVJGJYM7A, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.377735202 (587h45m27.248818247s) [noise=true , address=145.79.227.14:4837   , piece_id=23GZDADUIFN2OTV5BNPRMJ5MIKXJLRRDLOB5O5KDYOE3HJLGNIJA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.377896930 (587h45m27.248979975s) [noise=true , address=77.90.31.15:4700     , piece_id=L7K34LTYVM57KFEDJZEN7L2Y2HFAVQU5EDZXKIWS7ED5JFLDFZOQ, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.378408214 (587h45m27.249491259s) [noise=true , address=145.79.227.37:4565   , piece_id=YBOER4PYDEOSZZZIWMS6TYVIMFF34DMCQY45YHLSZISTM6AXOIFQ, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.378462918 (587h45m27.249545953s) [noise=true , address=45.91.116.11:4119    , piece_id=YEV2YPPCCLLMCHZY65ZMGU66U2JIPKCLIJCZCO3D45M7VSPLXYMA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.378821231 (587h45m27.249904276s) [noise=true , address=45.91.116.4:4939     , piece_id=FMI6IF3JBW6FOWU7VRQEFZCLSXTP6AY6ETVJLIEP5IQ3UPD5GNRQ, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.379250237 (587h45m27.250333282s) [noise=true , address=145.79.227.15:4876   , piece_id=5RMWWNADUHKU4EQCJ2ZZ5ZV45DN6BPGDPM7MLLEFDEVPHZYQYWQA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.379697038 (587h45m27.250780083s) [noise=true , address=45.91.116.5:4300     , piece_id=KU6Q4YAWNYHXQ4ERXL3W64ERC4B5OERR35BYPEIQ3FZEGPQIVXKA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.380171711 (587h45m27.251254746s) [noise=true , address=145.79.227.5:4221    , piece_id=N5TRSB7L23PL4N4KTRKFY7UEXEHQ5S77U6YVBLA4RK5MYVBQBN7Q, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.380555452 (587h45m27.251638497s) [noise=true , address=45.91.116.14:4190    , piece_id=62F4NHRQWURFCQRJSJ6O2LDKLRZVVNITWGE7IUEQDJBHGWSYF6YQ, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.381333154 (587h45m27.252416199s) [noise=true , address=45.91.116.8:4047     , piece_id=5AS6ME2MO6NKTLYQ6NFAE72POU6XGGGOAVTHJXCBLWK75LFA7DPA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.381746290 (587h45m27.252829335s) [noise=true , address=45.91.116.6:4973     , piece_id=GZ3R7DIUPHK4AXJBYQ5C5PBQU2DL25OI5PT4QNIKW7BAPY4BNGEA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.382331794 (587h45m27.25341484s) [noise=true , address=145.79.227.21:4475   , piece_id=ZFVBWT65I6QYW6U2CXSZZ4UWXZWLNAVHEJ5J6O5UR4UKPTJRZFAA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.382838620 (587h45m27.253921665s) [noise=true , address=77.90.31.14:4661     , piece_id=QXRYXOLIF4BRHJOPAWEMWRIZSYZAXGBKQCUHSZ3F3PWNRBKSTUMA, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:27.383097222 (587h45m27.254180267s) [noise=true , address=77.90.31.16:4736     , piece_id=UBRN5RFF7UOODV74N7HBN6RKSHWA4LRFCWROHH6UNSXQ4UDXDXTQ, seg_pos={0 0}, upload=515239]: Uploading piece...
Dec 30 03:14:29.082585888 (587h45m28.953668923s) [upload=515239]: Next returned nil segment
Dec 30 03:14:29.082599774 (587h45m28.953682809s) [upload=515239]: Waiting for error group managing segments...
Dec 30 03:14:29.356719242 (587h45m29.227802277s) [noise=true , address=145.79.227.14:4837   , piece_id=23GZDADUIFN2OTV5BNPRMJ5MIKXJLRRDLOB5O5KDYOE3HJLGNIJA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.361320505 (587h45m29.23240354s) [noise=true , address=145.79.227.15:4876   , piece_id=5RMWWNADUHKU4EQCJ2ZZ5ZV45DN6BPGDPM7MLLEFDEVPHZYQYWQA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.371841238 (587h45m29.242924303s) [noise=true , address=145.79.227.21:4475   , piece_id=ZFVBWT65I6QYW6U2CXSZZ4UWXZWLNAVHEJ5J6O5UR4UKPTJRZFAA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.377008917 (587h45m29.248091952s) [noise=true , address=145.79.227.5:4221    , piece_id=N5TRSB7L23PL4N4KTRKFY7UEXEHQ5S77U6YVBLA4RK5MYVBQBN7Q, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.380165918 (587h45m29.251248983s) [noise=true , address=145.79.227.46:4603   , piece_id=IEXLEOQE67PZTPA23JJIONBLH7XZGAQM5PBKQR6OGWZWVJGJYM7A, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.389672331 (587h45m29.260755366s) [noise=true , address=145.79.227.26:4654   , piece_id=6HME5K7VW5NCUPFCBRXDIHODOPECLPNWQJQ6I42OTQK3RBHLARIA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.607462068 (587h45m29.478545113s) [noise=true , address=45.91.116.4:4939     , piece_id=FMI6IF3JBW6FOWU7VRQEFZCLSXTP6AY6ETVJLIEP5IQ3UPD5GNRQ, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.643311411 (587h45m29.514394446s) [noise=true , address=45.91.116.8:4047     , piece_id=5AS6ME2MO6NKTLYQ6NFAE72POU6XGGGOAVTHJXCBLWK75LFA7DPA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.657020886 (587h45m29.528103921s) [noise=true , address=77.90.31.14:4661     , piece_id=QXRYXOLIF4BRHJOPAWEMWRIZSYZAXGBKQCUHSZ3F3PWNRBKSTUMA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.674864493 (587h45m29.545947528s) [noise=true , address=45.91.116.6:4973     , piece_id=GZ3R7DIUPHK4AXJBYQ5C5PBQU2DL25OI5PT4QNIKW7BAPY4BNGEA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.851103982 (587h45m29.722187027s) [noise=true , address=45.91.116.11:4119    , piece_id=YEV2YPPCCLLMCHZY65ZMGU66U2JIPKCLIJCZCO3D45M7VSPLXYMA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:29.972339214 (587h45m29.843422269s) [noise=true , address=77.90.31.15:4700     , piece_id=L7K34LTYVM57KFEDJZEN7L2Y2HFAVQU5EDZXKIWS7ED5JFLDFZOQ, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:30.321145789 (587h45m30.192228824s) [noise=true , address=77.90.31.16:4736     , piece_id=UBRN5RFF7UOODV74N7HBN6RKSHWA4LRFCWROHH6UNSXQ4UDXDXTQ, seg_pos={0 0}, upload=515239]: Done uploading piece. err: <nil>
Dec 30 03:14:30.321166298 (587h45m30.192249344s) [seg_pos={0 0}, upload=515239]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:14:30.321354677 (587h45m30.192437712s) [noise=true , address=45.91.116.14:4190    , piece_id=62F4NHRQWURFCQRJSJ6O2LDKLRZVVNITWGE7IUEQDJBHGWSYF6YQ, seg_pos={0 0}, upload=515239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF
Dec 30 03:14:30.321360648 (587h45m30.192443703s) [noise=true , address=45.91.116.14:4190    , piece_id=62F4NHRQWURFCQRJSJ6O2LDKLRZVVNITWGE7IUEQDJBHGWSYF6YQ, seg_pos={0 0}, upload=515239]: Long tail cancellation due to optimal threshold
Dec 30 03:14:30.321444969 (587h45m30.192528024s) [noise=true , address=145.79.227.37:4565   , piece_id=YBOER4PYDEOSZZZIWMS6TYVIMFF34DMCQY45YHLSZISTM6AXOIFQ, seg_pos={0 0}, upload=515239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:38988->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:38988->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->145.79.227.37:4565: use of closed network connection
Dec 30 03:14:30.321462823 (587h45m30.192545858s) [noise=true , address=145.79.227.37:4565   , piece_id=YBOER4PYDEOSZZZIWMS6TYVIMFF34DMCQY45YHLSZISTM6AXOIFQ, seg_pos={0 0}, upload=515239]: Long tail cancellation due to optimal threshold
Dec 30 03:14:30.321540551 (587h45m30.192623596s) [noise=true , address=45.91.116.5:4300     , piece_id=KU6Q4YAWNYHXQ4ERXL3W64ERC4B5OERR35BYPEIQ3FZEGPQIVXKA, seg_pos={0 0}, upload=515239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 30 03:14:30.321547324 (587h45m30.192630369s) [noise=true , address=45.91.116.5:4300     , piece_id=KU6Q4YAWNYHXQ4ERXL3W64ERC4B5OERR35BYPEIQ3FZEGPQIVXKA, seg_pos={0 0}, upload=515239]: Long tail cancellation due to optimal threshold
Dec 30 03:14:30.321563635 (587h45m30.19264667s) [seg_pos={0 0}, upload=515239]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:14:30.321631394 (587h45m30.192714439s) [upload=515239]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:14:30.321635712 (587h45m30.192718747s) [upload=515239]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:14:30.608229163 (587h45m30.479312218s) [upload=515239]: Done uploading segments...
Dec 30 03:14:30.608256105 (587h45m30.47933915s) [upload=515239]: Upload finished. err: <nil>
Duration: 30.002653063s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [.......................X...X.X.XX................XX.........]
duration: 
4.157649286s
trace: 
https://jaeger.dev.storj.io/trace/55b84c463909ac27
FAIL (2025-12-30 03:14:29 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.157704253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..........X................X......X.........................]
duration: 
48.907812512s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:14:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 48.907833833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [...X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X..]
duration: 
6.048839651s
trace: 
https://jaeger.dev.storj.io/trace/12186bf18ed51589
FAIL (2025-12-30 03:14:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 03:12:40.164940898 (587h43m40.036023933s) [upload=515213]: Starting upload
Dec 30 03:12:40.165050406 (587h43m40.036133461s) [upload=515213]: Done starting upload
Dec 30 03:12:40.165124849 (587h43m40.036207884s) [upload=515213]: Uploading segments...
Dec 30 03:12:40.165440490 (587h43m40.036523535s) [upload=515213]: Got next segment. Inline: false
Dec 30 03:12:40.165485375 (587h43m40.03656841s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:12:40.165491366 (587h43m40.036574401s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:12:40.418472013 (587h43m40.289555048s) [seg_pos={0 0}, upload=515213]: Begin upload segment...
Dec 30 03:12:40.418535334 (587h43m40.289618379s) [seg_pos={0 0}, upload=515213]: Done begin upload segment.
Dec 30 03:12:40.418582583 (587h43m40.289665618s) [noise=true , address=45.91.116.12:4228    , piece_id=IY5RKRFJJBPA2MX7C5ENRS47NWLYWKAFPKBLGK2ZFAC5QBVTNB5A, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.418650362 (587h43m40.289733407s) [noise=true , address=145.79.227.15:4876   , piece_id=TMFKJEE3B5I7KBW4UGUGL35QH7WJXY6PWCP4IAOY63DX4ZX6JXYQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.419272888 (587h43m40.290355923s) [noise=true , address=145.79.227.42:4123   , piece_id=WKWF4LQIQAIPMBTLPGSZKPN5DQTAWETH2JAUFF6XKKBP6BN3FX5Q, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.419752922 (587h43m40.290835967s) [noise=true , address=45.91.116.8:4048     , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.420289003 (587h43m40.291372048s) [noise=true , address=145.79.227.32:4836   , piece_id=6JBYTCQRY26XBMMJKJ4PNAWHSKZPPGWAQDOIDE7PNWQOZYBATJNQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.421578167 (587h43m40.292661212s) [noise=true , address=145.79.227.13:4803   , piece_id=BWOZMKEJOMP4FTJYEZAJLUTLM5W7BEQSB3NRMPVUGWILSJTTCRGA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.423655233 (587h43m40.294738278s) [noise=true , address=145.79.227.35:4482   , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.426910461 (587h43m40.297993506s) [noise=true , address=77.90.31.13:4626     , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.430745895 (587h43m40.30182896s) [noise=true , address=45.91.116.7:4011     , piece_id=ETL5GTRG2XJBIPJ4ACE2MC7SLBH3AA6BCUZMAI5ZHEWTV3WOQO7Q, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.430982735 (587h43m40.30206577s) [noise=true , address=145.79.227.48:4302   , piece_id=QGU5PJINPKFUOFORMBZFUYYKBWFZGAJT2LEMTWFLJVOWA34XGVJA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.432874588 (587h43m40.303957633s) [noise=true , address=145.79.227.14:4839   , piece_id=6LUFU5DX4V6P72BK2KFATJ42PJ75ZFQF6UOJYKYIHYDDILMX4AXQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.435392342 (587h43m40.306475377s) [noise=true , address=145.79.227.9:4367    , piece_id=BXRV2BDPTDB3GP2FGZNQKSCJ2VB25FSUOHQZXVC5LTY6YRJH2HBA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.436761078 (587h43m40.307844123s) [noise=true , address=45.91.116.10:4084    , piece_id=S7ELCREVROMSIU7SLSYEPFRE4POXIFP3JAH3G6LCKQ2Y5DTNCPFA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.442470289 (587h43m40.313553324s) [noise=true , address=145.79.227.2:4114    , piece_id=5CD5IHM2HKG4QS3ZD5YRS5EQTHA6KDRCXC2QM7J66LB4NXIPJQ7A, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.444326955 (587h43m40.31541s) [noise=true , address=145.79.227.19:4077   , piece_id=CGKRXP37UECASDCVQXZ4JKV3WXYVTZEBJHHAIO2DEZAKSWOQH22A, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.445001990 (587h43m40.316085025s) [noise=true , address=45.91.116.9:4263     , piece_id=LFRXNHFEP3QFDNOVXGSH6FOZC7S3DIMSG7ORTNBKHWX3S77ZSSMA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:42.118851413 (587h43m41.989934448s) [upload=515213]: Next returned nil segment
Dec 30 03:12:42.118863427 (587h43m41.989946472s) [upload=515213]: Waiting for error group managing segments...
Dec 30 03:12:42.405720249 (587h43m42.276803284s) [noise=true , address=145.79.227.32:4836   , piece_id=6JBYTCQRY26XBMMJKJ4PNAWHSKZPPGWAQDOIDE7PNWQOZYBATJNQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.411195465 (587h43m42.2822785s) [noise=true , address=145.79.227.42:4123   , piece_id=WKWF4LQIQAIPMBTLPGSZKPN5DQTAWETH2JAUFF6XKKBP6BN3FX5Q, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.415202575 (587h43m42.28628561s) [noise=true , address=145.79.227.13:4803   , piece_id=BWOZMKEJOMP4FTJYEZAJLUTLM5W7BEQSB3NRMPVUGWILSJTTCRGA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.420472730 (587h43m42.291555795s) [noise=true , address=145.79.227.9:4367    , piece_id=BXRV2BDPTDB3GP2FGZNQKSCJ2VB25FSUOHQZXVC5LTY6YRJH2HBA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.446510851 (587h43m42.317593896s) [noise=true , address=145.79.227.15:4876   , piece_id=TMFKJEE3B5I7KBW4UGUGL35QH7WJXY6PWCP4IAOY63DX4ZX6JXYQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.447650281 (587h43m42.318733326s) [noise=true , address=145.79.227.19:4077   , piece_id=CGKRXP37UECASDCVQXZ4JKV3WXYVTZEBJHHAIO2DEZAKSWOQH22A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.448275281 (587h43m42.319358326s) [noise=true , address=145.79.227.48:4302   , piece_id=QGU5PJINPKFUOFORMBZFUYYKBWFZGAJT2LEMTWFLJVOWA34XGVJA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.453726491 (587h43m42.324809526s) [noise=true , address=145.79.227.14:4839   , piece_id=6LUFU5DX4V6P72BK2KFATJ42PJ75ZFQF6UOJYKYIHYDDILMX4AXQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.456498640 (587h43m42.327581675s) [noise=true , address=145.79.227.2:4114    , piece_id=5CD5IHM2HKG4QS3ZD5YRS5EQTHA6KDRCXC2QM7J66LB4NXIPJQ7A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.633626620 (587h43m42.504709666s) [noise=true , address=45.91.116.10:4084    , piece_id=S7ELCREVROMSIU7SLSYEPFRE4POXIFP3JAH3G6LCKQ2Y5DTNCPFA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.689063291 (587h43m42.560146326s) [noise=true , address=45.91.116.12:4228    , piece_id=IY5RKRFJJBPA2MX7C5ENRS47NWLYWKAFPKBLGK2ZFAC5QBVTNB5A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.838513469 (587h43m42.709596504s) [noise=true , address=45.91.116.7:4011     , piece_id=ETL5GTRG2XJBIPJ4ACE2MC7SLBH3AA6BCUZMAI5ZHEWTV3WOQO7Q, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.996672750 (587h43m42.867755785s) [noise=true , address=45.91.116.9:4263     , piece_id=LFRXNHFEP3QFDNOVXGSH6FOZC7S3DIMSG7ORTNBKHWX3S77ZSSMA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.996690483 (587h43m42.867773518s) [seg_pos={0 0}, upload=515213]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:12:42.996877650 (587h43m42.867960685s) [noise=true , address=145.79.227.35:4482   , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 03:12:42.996885345 (587h43m42.86796838s) [noise=true , address=145.79.227.35:4482   , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold
Dec 30 03:12:42.996999732 (587h43m42.868082777s) [noise=true , address=77.90.31.13:4626     , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection
Dec 30 03:12:42.997010552 (587h43m42.868093597s) [noise=true , address=77.90.31.13:4626     , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold
Dec 30 03:12:42.997111405 (587h43m42.86819444s) [noise=true , address=45.91.116.8:4048     , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 30 03:12:42.997117786 (587h43m42.868200822s) [noise=true , address=45.91.116.8:4048     , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold
Dec 30 03:12:42.997133918 (587h43m42.868216963s) [seg_pos={0 0}, upload=515213]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:12:42.997261691 (587h43m42.868344726s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:12:42.997268534 (587h43m42.868351579s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:12:43.280439328 (587h43m43.151522373s) [upload=515213]: Done uploading segments...
Dec 30 03:12:43.280462953 (587h43m43.151545998s) [upload=515213]: Upload finished. err: <nil>
Duration: 6.048867054s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [........................X...X.X.XX................XX........]
duration: 
3.543450592s
trace: 
https://jaeger.dev.storj.io/trace/55b84c463909ac27
FAIL (2025-12-30 03:13:40 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.543470229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [....X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X.]
duration: 
6.177931737s
trace: 
https://jaeger.dev.storj.io/trace/12186bf18ed51589
FAIL (2025-12-30 03:13:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 03:12:40.164940898 (587h43m40.036023933s) [upload=515213]: Starting upload
Dec 30 03:12:40.165050406 (587h43m40.036133461s) [upload=515213]: Done starting upload
Dec 30 03:12:40.165124849 (587h43m40.036207884s) [upload=515213]: Uploading segments...
Dec 30 03:12:40.165440490 (587h43m40.036523535s) [upload=515213]: Got next segment. Inline: false
Dec 30 03:12:40.165485375 (587h43m40.03656841s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:12:40.165491366 (587h43m40.036574401s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:12:40.418472013 (587h43m40.289555048s) [seg_pos={0 0}, upload=515213]: Begin upload segment...
Dec 30 03:12:40.418535334 (587h43m40.289618379s) [seg_pos={0 0}, upload=515213]: Done begin upload segment.
Dec 30 03:12:40.418582583 (587h43m40.289665618s) [noise=true , address=45.91.116.12:4228    , piece_id=IY5RKRFJJBPA2MX7C5ENRS47NWLYWKAFPKBLGK2ZFAC5QBVTNB5A, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.418650362 (587h43m40.289733407s) [noise=true , address=145.79.227.15:4876   , piece_id=TMFKJEE3B5I7KBW4UGUGL35QH7WJXY6PWCP4IAOY63DX4ZX6JXYQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.419272888 (587h43m40.290355923s) [noise=true , address=145.79.227.42:4123   , piece_id=WKWF4LQIQAIPMBTLPGSZKPN5DQTAWETH2JAUFF6XKKBP6BN3FX5Q, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.419752922 (587h43m40.290835967s) [noise=true , address=45.91.116.8:4048     , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.420289003 (587h43m40.291372048s) [noise=true , address=145.79.227.32:4836   , piece_id=6JBYTCQRY26XBMMJKJ4PNAWHSKZPPGWAQDOIDE7PNWQOZYBATJNQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.421578167 (587h43m40.292661212s) [noise=true , address=145.79.227.13:4803   , piece_id=BWOZMKEJOMP4FTJYEZAJLUTLM5W7BEQSB3NRMPVUGWILSJTTCRGA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.423655233 (587h43m40.294738278s) [noise=true , address=145.79.227.35:4482   , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.426910461 (587h43m40.297993506s) [noise=true , address=77.90.31.13:4626     , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.430745895 (587h43m40.30182896s) [noise=true , address=45.91.116.7:4011     , piece_id=ETL5GTRG2XJBIPJ4ACE2MC7SLBH3AA6BCUZMAI5ZHEWTV3WOQO7Q, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.430982735 (587h43m40.30206577s) [noise=true , address=145.79.227.48:4302   , piece_id=QGU5PJINPKFUOFORMBZFUYYKBWFZGAJT2LEMTWFLJVOWA34XGVJA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.432874588 (587h43m40.303957633s) [noise=true , address=145.79.227.14:4839   , piece_id=6LUFU5DX4V6P72BK2KFATJ42PJ75ZFQF6UOJYKYIHYDDILMX4AXQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.435392342 (587h43m40.306475377s) [noise=true , address=145.79.227.9:4367    , piece_id=BXRV2BDPTDB3GP2FGZNQKSCJ2VB25FSUOHQZXVC5LTY6YRJH2HBA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.436761078 (587h43m40.307844123s) [noise=true , address=45.91.116.10:4084    , piece_id=S7ELCREVROMSIU7SLSYEPFRE4POXIFP3JAH3G6LCKQ2Y5DTNCPFA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.442470289 (587h43m40.313553324s) [noise=true , address=145.79.227.2:4114    , piece_id=5CD5IHM2HKG4QS3ZD5YRS5EQTHA6KDRCXC2QM7J66LB4NXIPJQ7A, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.444326955 (587h43m40.31541s) [noise=true , address=145.79.227.19:4077   , piece_id=CGKRXP37UECASDCVQXZ4JKV3WXYVTZEBJHHAIO2DEZAKSWOQH22A, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.445001990 (587h43m40.316085025s) [noise=true , address=45.91.116.9:4263     , piece_id=LFRXNHFEP3QFDNOVXGSH6FOZC7S3DIMSG7ORTNBKHWX3S77ZSSMA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:42.118851413 (587h43m41.989934448s) [upload=515213]: Next returned nil segment
Dec 30 03:12:42.118863427 (587h43m41.989946472s) [upload=515213]: Waiting for error group managing segments...
Dec 30 03:12:42.405720249 (587h43m42.276803284s) [noise=true , address=145.79.227.32:4836   , piece_id=6JBYTCQRY26XBMMJKJ4PNAWHSKZPPGWAQDOIDE7PNWQOZYBATJNQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.411195465 (587h43m42.2822785s) [noise=true , address=145.79.227.42:4123   , piece_id=WKWF4LQIQAIPMBTLPGSZKPN5DQTAWETH2JAUFF6XKKBP6BN3FX5Q, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.415202575 (587h43m42.28628561s) [noise=true , address=145.79.227.13:4803   , piece_id=BWOZMKEJOMP4FTJYEZAJLUTLM5W7BEQSB3NRMPVUGWILSJTTCRGA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.420472730 (587h43m42.291555795s) [noise=true , address=145.79.227.9:4367    , piece_id=BXRV2BDPTDB3GP2FGZNQKSCJ2VB25FSUOHQZXVC5LTY6YRJH2HBA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.446510851 (587h43m42.317593896s) [noise=true , address=145.79.227.15:4876   , piece_id=TMFKJEE3B5I7KBW4UGUGL35QH7WJXY6PWCP4IAOY63DX4ZX6JXYQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.447650281 (587h43m42.318733326s) [noise=true , address=145.79.227.19:4077   , piece_id=CGKRXP37UECASDCVQXZ4JKV3WXYVTZEBJHHAIO2DEZAKSWOQH22A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.448275281 (587h43m42.319358326s) [noise=true , address=145.79.227.48:4302   , piece_id=QGU5PJINPKFUOFORMBZFUYYKBWFZGAJT2LEMTWFLJVOWA34XGVJA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.453726491 (587h43m42.324809526s) [noise=true , address=145.79.227.14:4839   , piece_id=6LUFU5DX4V6P72BK2KFATJ42PJ75ZFQF6UOJYKYIHYDDILMX4AXQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.456498640 (587h43m42.327581675s) [noise=true , address=145.79.227.2:4114    , piece_id=5CD5IHM2HKG4QS3ZD5YRS5EQTHA6KDRCXC2QM7J66LB4NXIPJQ7A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.633626620 (587h43m42.504709666s) [noise=true , address=45.91.116.10:4084    , piece_id=S7ELCREVROMSIU7SLSYEPFRE4POXIFP3JAH3G6LCKQ2Y5DTNCPFA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.689063291 (587h43m42.560146326s) [noise=true , address=45.91.116.12:4228    , piece_id=IY5RKRFJJBPA2MX7C5ENRS47NWLYWKAFPKBLGK2ZFAC5QBVTNB5A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.838513469 (587h43m42.709596504s) [noise=true , address=45.91.116.7:4011     , piece_id=ETL5GTRG2XJBIPJ4ACE2MC7SLBH3AA6BCUZMAI5ZHEWTV3WOQO7Q, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.996672750 (587h43m42.867755785s) [noise=true , address=45.91.116.9:4263     , piece_id=LFRXNHFEP3QFDNOVXGSH6FOZC7S3DIMSG7ORTNBKHWX3S77ZSSMA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.996690483 (587h43m42.867773518s) [seg_pos={0 0}, upload=515213]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:12:42.996877650 (587h43m42.867960685s) [noise=true , address=145.79.227.35:4482   , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 03:12:42.996885345 (587h43m42.86796838s) [noise=true , address=145.79.227.35:4482   , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold
Dec 30 03:12:42.996999732 (587h43m42.868082777s) [noise=true , address=77.90.31.13:4626     , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection
Dec 30 03:12:42.997010552 (587h43m42.868093597s) [noise=true , address=77.90.31.13:4626     , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold
Dec 30 03:12:42.997111405 (587h43m42.86819444s) [noise=true , address=45.91.116.8:4048     , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 30 03:12:42.997117786 (587h43m42.868200822s) [noise=true , address=45.91.116.8:4048     , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold
Dec 30 03:12:42.997133918 (587h43m42.868216963s) [seg_pos={0 0}, upload=515213]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:12:42.997261691 (587h43m42.868344726s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:12:42.997268534 (587h43m42.868351579s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:12:43.280439328 (587h43m43.151522373s) [upload=515213]: Done uploading segments...
Dec 30 03:12:43.280462953 (587h43m43.151545998s) [upload=515213]: Upload finished. err: <nil>
Duration: 6.177951023s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...........X................X......X........................]
duration: 
43.723074994s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:13:14 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 43.723101745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.....X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X]
duration: 
30.00593005s
trace: 
https://jaeger.dev.storj.io/trace/12186bf18ed51589
FAIL (2025-12-30 03:13:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 03:12:40.164940898 (587h43m40.036023933s) [upload=515213]: Starting upload
Dec 30 03:12:40.165050406 (587h43m40.036133461s) [upload=515213]: Done starting upload
Dec 30 03:12:40.165124849 (587h43m40.036207884s) [upload=515213]: Uploading segments...
Dec 30 03:12:40.165440490 (587h43m40.036523535s) [upload=515213]: Got next segment. Inline: false
Dec 30 03:12:40.165485375 (587h43m40.03656841s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 03:12:40.165491366 (587h43m40.036574401s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 03:12:40.418472013 (587h43m40.289555048s) [seg_pos={0 0}, upload=515213]: Begin upload segment...
Dec 30 03:12:40.418535334 (587h43m40.289618379s) [seg_pos={0 0}, upload=515213]: Done begin upload segment.
Dec 30 03:12:40.418582583 (587h43m40.289665618s) [noise=true , address=45.91.116.12:4228    , piece_id=IY5RKRFJJBPA2MX7C5ENRS47NWLYWKAFPKBLGK2ZFAC5QBVTNB5A, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.418650362 (587h43m40.289733407s) [noise=true , address=145.79.227.15:4876   , piece_id=TMFKJEE3B5I7KBW4UGUGL35QH7WJXY6PWCP4IAOY63DX4ZX6JXYQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.419272888 (587h43m40.290355923s) [noise=true , address=145.79.227.42:4123   , piece_id=WKWF4LQIQAIPMBTLPGSZKPN5DQTAWETH2JAUFF6XKKBP6BN3FX5Q, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.419752922 (587h43m40.290835967s) [noise=true , address=45.91.116.8:4048     , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.420289003 (587h43m40.291372048s) [noise=true , address=145.79.227.32:4836   , piece_id=6JBYTCQRY26XBMMJKJ4PNAWHSKZPPGWAQDOIDE7PNWQOZYBATJNQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.421578167 (587h43m40.292661212s) [noise=true , address=145.79.227.13:4803   , piece_id=BWOZMKEJOMP4FTJYEZAJLUTLM5W7BEQSB3NRMPVUGWILSJTTCRGA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.423655233 (587h43m40.294738278s) [noise=true , address=145.79.227.35:4482   , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.426910461 (587h43m40.297993506s) [noise=true , address=77.90.31.13:4626     , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.430745895 (587h43m40.30182896s) [noise=true , address=45.91.116.7:4011     , piece_id=ETL5GTRG2XJBIPJ4ACE2MC7SLBH3AA6BCUZMAI5ZHEWTV3WOQO7Q, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.430982735 (587h43m40.30206577s) [noise=true , address=145.79.227.48:4302   , piece_id=QGU5PJINPKFUOFORMBZFUYYKBWFZGAJT2LEMTWFLJVOWA34XGVJA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.432874588 (587h43m40.303957633s) [noise=true , address=145.79.227.14:4839   , piece_id=6LUFU5DX4V6P72BK2KFATJ42PJ75ZFQF6UOJYKYIHYDDILMX4AXQ, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.435392342 (587h43m40.306475377s) [noise=true , address=145.79.227.9:4367    , piece_id=BXRV2BDPTDB3GP2FGZNQKSCJ2VB25FSUOHQZXVC5LTY6YRJH2HBA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.436761078 (587h43m40.307844123s) [noise=true , address=45.91.116.10:4084    , piece_id=S7ELCREVROMSIU7SLSYEPFRE4POXIFP3JAH3G6LCKQ2Y5DTNCPFA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.442470289 (587h43m40.313553324s) [noise=true , address=145.79.227.2:4114    , piece_id=5CD5IHM2HKG4QS3ZD5YRS5EQTHA6KDRCXC2QM7J66LB4NXIPJQ7A, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.444326955 (587h43m40.31541s) [noise=true , address=145.79.227.19:4077   , piece_id=CGKRXP37UECASDCVQXZ4JKV3WXYVTZEBJHHAIO2DEZAKSWOQH22A, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:40.445001990 (587h43m40.316085025s) [noise=true , address=45.91.116.9:4263     , piece_id=LFRXNHFEP3QFDNOVXGSH6FOZC7S3DIMSG7ORTNBKHWX3S77ZSSMA, seg_pos={0 0}, upload=515213]: Uploading piece...
Dec 30 03:12:42.118851413 (587h43m41.989934448s) [upload=515213]: Next returned nil segment
Dec 30 03:12:42.118863427 (587h43m41.989946472s) [upload=515213]: Waiting for error group managing segments...
Dec 30 03:12:42.405720249 (587h43m42.276803284s) [noise=true , address=145.79.227.32:4836   , piece_id=6JBYTCQRY26XBMMJKJ4PNAWHSKZPPGWAQDOIDE7PNWQOZYBATJNQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.411195465 (587h43m42.2822785s) [noise=true , address=145.79.227.42:4123   , piece_id=WKWF4LQIQAIPMBTLPGSZKPN5DQTAWETH2JAUFF6XKKBP6BN3FX5Q, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.415202575 (587h43m42.28628561s) [noise=true , address=145.79.227.13:4803   , piece_id=BWOZMKEJOMP4FTJYEZAJLUTLM5W7BEQSB3NRMPVUGWILSJTTCRGA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.420472730 (587h43m42.291555795s) [noise=true , address=145.79.227.9:4367    , piece_id=BXRV2BDPTDB3GP2FGZNQKSCJ2VB25FSUOHQZXVC5LTY6YRJH2HBA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.446510851 (587h43m42.317593896s) [noise=true , address=145.79.227.15:4876   , piece_id=TMFKJEE3B5I7KBW4UGUGL35QH7WJXY6PWCP4IAOY63DX4ZX6JXYQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.447650281 (587h43m42.318733326s) [noise=true , address=145.79.227.19:4077   , piece_id=CGKRXP37UECASDCVQXZ4JKV3WXYVTZEBJHHAIO2DEZAKSWOQH22A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.448275281 (587h43m42.319358326s) [noise=true , address=145.79.227.48:4302   , piece_id=QGU5PJINPKFUOFORMBZFUYYKBWFZGAJT2LEMTWFLJVOWA34XGVJA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.453726491 (587h43m42.324809526s) [noise=true , address=145.79.227.14:4839   , piece_id=6LUFU5DX4V6P72BK2KFATJ42PJ75ZFQF6UOJYKYIHYDDILMX4AXQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.456498640 (587h43m42.327581675s) [noise=true , address=145.79.227.2:4114    , piece_id=5CD5IHM2HKG4QS3ZD5YRS5EQTHA6KDRCXC2QM7J66LB4NXIPJQ7A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.633626620 (587h43m42.504709666s) [noise=true , address=45.91.116.10:4084    , piece_id=S7ELCREVROMSIU7SLSYEPFRE4POXIFP3JAH3G6LCKQ2Y5DTNCPFA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.689063291 (587h43m42.560146326s) [noise=true , address=45.91.116.12:4228    , piece_id=IY5RKRFJJBPA2MX7C5ENRS47NWLYWKAFPKBLGK2ZFAC5QBVTNB5A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.838513469 (587h43m42.709596504s) [noise=true , address=45.91.116.7:4011     , piece_id=ETL5GTRG2XJBIPJ4ACE2MC7SLBH3AA6BCUZMAI5ZHEWTV3WOQO7Q, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.996672750 (587h43m42.867755785s) [noise=true , address=45.91.116.9:4263     , piece_id=LFRXNHFEP3QFDNOVXGSH6FOZC7S3DIMSG7ORTNBKHWX3S77ZSSMA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil>
Dec 30 03:12:42.996690483 (587h43m42.867773518s) [seg_pos={0 0}, upload=515213]: Segment reached optimal threshold of 13 pieces.
Dec 30 03:12:42.996877650 (587h43m42.867960685s) [noise=true , address=145.79.227.35:4482   , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 03:12:42.996885345 (587h43m42.86796838s) [noise=true , address=145.79.227.35:4482   , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold
Dec 30 03:12:42.996999732 (587h43m42.868082777s) [noise=true , address=77.90.31.13:4626     , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection
Dec 30 03:12:42.997010552 (587h43m42.868093597s) [noise=true , address=77.90.31.13:4626     , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold
Dec 30 03:12:42.997111405 (587h43m42.86819444s) [noise=true , address=45.91.116.8:4048     , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 30 03:12:42.997117786 (587h43m42.868200822s) [noise=true , address=45.91.116.8:4048     , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold
Dec 30 03:12:42.997133918 (587h43m42.868216963s) [seg_pos={0 0}, upload=515213]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 03:12:42.997261691 (587h43m42.868344726s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 03:12:42.997268534 (587h43m42.868351579s) [upload=515213]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 03:12:43.280439328 (587h43m43.151522373s) [upload=515213]: Done uploading segments...
Dec 30 03:12:43.280462953 (587h43m43.151545998s) [upload=515213]: Upload finished. err: <nil>
Duration: 30.005946041s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [.........................X...X.X.XX................XX.......]
duration: 
5.000742623s
trace: 
https://jaeger.dev.storj.io/trace/55b84c463909ac27
FAIL (2025-12-30 03:12:54 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000766839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [............X................X......X.......................]
duration: 
32.687059968s
trace: 
https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9
FAIL (2025-12-30 03:12:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece...
Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment
Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments...
Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959     , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837   , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674   , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623   , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496   , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472   , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881     , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573     , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104    , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601     , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309     , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885   , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187   , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053   , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil>
Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces.
Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780   , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection
Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429   , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176    , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340   , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold
Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments...
Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil>
Duration: 32.687089434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [..........................X...X.X.XX................XX......]
duration: 
3.661545485s
trace: 
https://jaeger.dev.storj.io/trace/55b84c463909ac27
FAIL (2025-12-30 03:12:11 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.661566053s core uplink us-select-1 us1-placement-20