Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX........X.....X.............X.............................] duration: 33.620513173s trace: https://jaeger.dev.storj.io/trace/5943c4171683f71a FAIL (2025-12-31 04:53:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments... Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces. Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments... Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil> Duration: 33.620539254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.13839013s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:53:26 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.138421289s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 11.290007017s trace: https://jaeger.dev.storj.io/trace/1080f2dcf8b139c9 FAIL (2025-12-31 04:53:24 UTC) orbiter warning: context deadline exceeded Duration: 11.290043537s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.023629971s trace: https://jaeger.dev.storj.io/trace/1080f2dcf8b139c9 FAIL (2025-12-31 04:53:03 UTC) orbiter warning: context deadline exceeded Duration: 7.02365641s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......XX.X...X...X..X...X.......X.......X.........XX...X.X.] duration: 5.808482358s trace: https://jaeger.dev.storj.io/trace/12ce8f0793bb0cf5 FAIL (2025-12-31 04:52:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: S4GCNZNHKVUOJKXXRJQDP5I6FMRFJAXJKHHF3CZJL2R7FD2PM4HQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KOWJSZ4QJ7I74LJ56DNJIWIEVG2PYDTMR5GV7YGNOVBN6N4E5BQQ) context deadline exceeded ======LOGS====== Dec 31 04:52:21.633896208 (613h23m21.504979253s) [upload=537532]: Starting upload Dec 31 04:52:21.633957856 (613h23m21.505040901s) [upload=537532]: Done starting upload Dec 31 04:52:21.634044180 (613h23m21.505127215s) [upload=537532]: Uploading segments... Dec 31 04:52:21.634168327 (613h23m21.505251372s) [upload=537532]: Got next segment. Inline: false Dec 31 04:52:21.634212030 (613h23m21.505295065s) [upload=537532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:52:21.634219674 (613h23m21.505302709s) [upload=537532]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:52:21.878649529 (613h23m21.749732574s) [seg_pos={0 0}, upload=537532]: Begin upload segment... Dec 31 04:52:21.878695166 (613h23m21.749778201s) [seg_pos={0 0}, upload=537532]: Done begin upload segment. Dec 31 04:52:21.878725123 (613h23m21.749808168s) [noise=true , address=145.79.227.10:4403 , piece_id=PPIXOQODOH6NNBPKU6PNOB764ELLAC7QFALF7QSF5C4TV2532WWQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.878923391 (613h23m21.750006436s) [noise=true , address=145.79.227.13:4803 , piece_id=EQBPPBHGTHIJXLQUH45DL2M3DXVMZU736F6YREYGXAYMLJYDQFSA, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.879962258 (613h23m21.751045293s) [noise=true , address=145.79.227.9:4366 , piece_id=OQ76IWMTLCX6LTUJMR5CDMHC3WOS742EPOSZZ7M23LKIASJHQ3YQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.880497017 (613h23m21.751580062s) [noise=true , address=145.79.227.21:4473 , piece_id=VFSECHPXN7LZVVPX7JBOYYHASVNZZ6CIOZA5SLOA7SEYTQKCZNFQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.880576788 (613h23m21.751659824s) [noise=true , address=145.79.227.22:4511 , piece_id=VHEIWH7UQUPWQKOYZLJXOHYXIRVQTTNITSE5S6PCHEUMPJNTA4PA, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.881771673 (613h23m21.752854708s) [noise=true , address=77.90.31.19:4844 , piece_id=S4GCNZNHKVUOJKXXRJQDP5I6FMRFJAXJKHHF3CZJL2R7FD2PM4HQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.882586725 (613h23m21.75366977s) [noise=true , address=77.90.31.11:4555 , piece_id=HH2B3AGQFXV6VIHSXL5SBEICYIIRRJAWQPLFQKDUE2Y2OJYO6JEQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.882727344 (613h23m21.753810389s) [noise=true , address=145.79.227.37:4565 , piece_id=KLPKMCT3OW3ZWLFNPDLUZJKO5ZYLHGDHLZ6T3MD2QRFRUNHRQYEA, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.883240861 (613h23m21.754323896s) [noise=true , address=145.79.227.41:4085 , piece_id=JNEAUJ5UHTU7QNX3VGUUI262MF5WT63V6YSMVR4ZFBT3MCURB4GA, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.883613911 (613h23m21.754696957s) [noise=true , address=45.91.116.10:4082 , piece_id=CCXTESJT76SQ2UKBAZPJXKRWBWYUYANLJFARBQ3N24QTBY5DUP6A, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.884089247 (613h23m21.755172292s) [noise=true , address=77.90.31.12:4589 , piece_id=KOWJSZ4QJ7I74LJ56DNJIWIEVG2PYDTMR5GV7YGNOVBN6N4E5BQQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.884635176 (613h23m21.755718211s) [noise=true , address=145.79.227.4:4186 , piece_id=A6OISTIMY3RT5JWUG6OV6OTBT235O5AE3H24DLV7KITBXFQZY4ZQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.884975454 (613h23m21.756058489s) [noise=true , address=145.79.227.16:4909 , piece_id=T2NPZNHXE426UCEYBAVIUPTOGZQTHLBHAHMSZCHUFPYU5EOSVOZQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.885934921 (613h23m21.757017956s) [noise=true , address=45.91.116.13:4155 , piece_id=RVOYY7MEK2QR4GQGIG3RZKMFUSNQ5H7BXKNOGQ4BZPFKAJLO6OBA, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.887162489 (613h23m21.758245534s) [noise=true , address=145.79.227.46:4601 , piece_id=JO2LVBIKZH4HXU5HSWIQMJDOT52M6RU4B7AC4AJC4RZFG573RCMQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.887729949 (613h23m21.758812984s) [noise=true , address=145.79.227.39:4015 , piece_id=YAUFJBFWHOZRAFMMU3JBVDGA2JYM2KLCW63PBV6SQJDGMNDD6P4Q, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:23.625922551 (613h23m23.497005596s) [upload=537532]: Next returned nil segment Dec 31 04:52:23.625937369 (613h23m23.497020404s) [upload=537532]: Waiting for error group managing segments... Dec 31 04:52:23.881727836 (613h23m23.752810871s) [noise=true , address=145.79.227.21:4473 , piece_id=VFSECHPXN7LZVVPX7JBOYYHASVNZZ6CIOZA5SLOA7SEYTQKCZNFQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.883922876 (613h23m23.755005921s) [noise=true , address=145.79.227.9:4366 , piece_id=OQ76IWMTLCX6LTUJMR5CDMHC3WOS742EPOSZZ7M23LKIASJHQ3YQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.886029607 (613h23m23.757112642s) [noise=true , address=145.79.227.16:4909 , piece_id=T2NPZNHXE426UCEYBAVIUPTOGZQTHLBHAHMSZCHUFPYU5EOSVOZQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.890565283 (613h23m23.761648328s) [noise=true , address=145.79.227.4:4186 , piece_id=A6OISTIMY3RT5JWUG6OV6OTBT235O5AE3H24DLV7KITBXFQZY4ZQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.893963995 (613h23m23.76504704s) [noise=true , address=145.79.227.39:4015 , piece_id=YAUFJBFWHOZRAFMMU3JBVDGA2JYM2KLCW63PBV6SQJDGMNDD6P4Q, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.894813392 (613h23m23.765896427s) [noise=true , address=145.79.227.41:4085 , piece_id=JNEAUJ5UHTU7QNX3VGUUI262MF5WT63V6YSMVR4ZFBT3MCURB4GA, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.895110198 (613h23m23.766193233s) [noise=true , address=145.79.227.10:4403 , piece_id=PPIXOQODOH6NNBPKU6PNOB764ELLAC7QFALF7QSF5C4TV2532WWQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.937499191 (613h23m23.808582236s) [noise=true , address=145.79.227.13:4803 , piece_id=EQBPPBHGTHIJXLQUH45DL2M3DXVMZU736F6YREYGXAYMLJYDQFSA, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.940935795 (613h23m23.812018841s) [noise=true , address=145.79.227.46:4601 , piece_id=JO2LVBIKZH4HXU5HSWIQMJDOT52M6RU4B7AC4AJC4RZFG573RCMQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.963207784 (613h23m23.834290819s) [noise=true , address=145.79.227.37:4565 , piece_id=KLPKMCT3OW3ZWLFNPDLUZJKO5ZYLHGDHLZ6T3MD2QRFRUNHRQYEA, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:24.041218496 (613h23m23.912301531s) [noise=true , address=145.79.227.22:4511 , piece_id=VHEIWH7UQUPWQKOYZLJXOHYXIRVQTTNITSE5S6PCHEUMPJNTA4PA, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:24.158777742 (613h23m24.029860787s) [noise=true , address=77.90.31.12:4589 , piece_id=KOWJSZ4QJ7I74LJ56DNJIWIEVG2PYDTMR5GV7YGNOVBN6N4E5BQQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:24.187116573 (613h23m24.058199628s) [noise=true , address=77.90.31.19:4844 , piece_id=S4GCNZNHKVUOJKXXRJQDP5I6FMRFJAXJKHHF3CZJL2R7FD2PM4HQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:24.187143584 (613h23m24.058226619s) [seg_pos={0 0}, upload=537532]: Segment reached optimal threshold of 13 pieces. Dec 31 04:52:24.187566188 (613h23m24.058649223s) [noise=true , address=45.91.116.13:4155 , piece_id=RVOYY7MEK2QR4GQGIG3RZKMFUSNQ5H7BXKNOGQ4BZPFKAJLO6OBA, seg_pos={0 0}, upload=537532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 04:52:24.187611756 (613h23m24.058694791s) [noise=true , address=45.91.116.13:4155 , piece_id=RVOYY7MEK2QR4GQGIG3RZKMFUSNQ5H7BXKNOGQ4BZPFKAJLO6OBA, seg_pos={0 0}, upload=537532]: Long tail cancellation due to optimal threshold Dec 31 04:52:24.187792911 (613h23m24.058875966s) [noise=true , address=77.90.31.11:4555 , piece_id=HH2B3AGQFXV6VIHSXL5SBEICYIIRRJAWQPLFQKDUE2Y2OJYO6JEQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 31 04:52:24.187806506 (613h23m24.058889541s) [noise=true , address=77.90.31.11:4555 , piece_id=HH2B3AGQFXV6VIHSXL5SBEICYIIRRJAWQPLFQKDUE2Y2OJYO6JEQ, seg_pos={0 0}, upload=537532]: Long tail cancellation due to optimal threshold Dec 31 04:52:24.188023861 (613h23m24.059106906s) [noise=true , address=45.91.116.10:4082 , piece_id=CCXTESJT76SQ2UKBAZPJXKRWBWYUYANLJFARBQ3N24QTBY5DUP6A, seg_pos={0 0}, upload=537532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 31 04:52:24.188049659 (613h23m24.059132694s) [noise=true , address=45.91.116.10:4082 , piece_id=CCXTESJT76SQ2UKBAZPJXKRWBWYUYANLJFARBQ3N24QTBY5DUP6A, seg_pos={0 0}, upload=537532]: Long tail cancellation due to optimal threshold Dec 31 04:52:24.188072353 (613h23m24.059155388s) [seg_pos={0 0}, upload=537532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:52:24.188170530 (613h23m24.059253575s) [upload=537532]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:52:24.188176060 (613h23m24.059259095s) [upload=537532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:52:24.475797317 (613h23m24.346880362s) [upload=537532]: Done uploading segments... Dec 31 04:52:24.475819960 (613h23m24.346903005s) [upload=537532]: Upload finished. err: <nil> Duration: 5.808496976s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........XX.X...X...X..X...X.......X.......X.........XX...X.X] duration: 30.002954387s trace: https://jaeger.dev.storj.io/trace/12ce8f0793bb0cf5 FAIL (2025-12-31 04:52:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: S4GCNZNHKVUOJKXXRJQDP5I6FMRFJAXJKHHF3CZJL2R7FD2PM4HQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KOWJSZ4QJ7I74LJ56DNJIWIEVG2PYDTMR5GV7YGNOVBN6N4E5BQQ) context deadline exceeded ======LOGS====== Dec 31 04:52:21.633896208 (613h23m21.504979253s) [upload=537532]: Starting upload Dec 31 04:52:21.633957856 (613h23m21.505040901s) [upload=537532]: Done starting upload Dec 31 04:52:21.634044180 (613h23m21.505127215s) [upload=537532]: Uploading segments... Dec 31 04:52:21.634168327 (613h23m21.505251372s) [upload=537532]: Got next segment. Inline: false Dec 31 04:52:21.634212030 (613h23m21.505295065s) [upload=537532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:52:21.634219674 (613h23m21.505302709s) [upload=537532]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:52:21.878649529 (613h23m21.749732574s) [seg_pos={0 0}, upload=537532]: Begin upload segment... Dec 31 04:52:21.878695166 (613h23m21.749778201s) [seg_pos={0 0}, upload=537532]: Done begin upload segment. Dec 31 04:52:21.878725123 (613h23m21.749808168s) [noise=true , address=145.79.227.10:4403 , piece_id=PPIXOQODOH6NNBPKU6PNOB764ELLAC7QFALF7QSF5C4TV2532WWQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.878923391 (613h23m21.750006436s) [noise=true , address=145.79.227.13:4803 , piece_id=EQBPPBHGTHIJXLQUH45DL2M3DXVMZU736F6YREYGXAYMLJYDQFSA, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.879962258 (613h23m21.751045293s) [noise=true , address=145.79.227.9:4366 , piece_id=OQ76IWMTLCX6LTUJMR5CDMHC3WOS742EPOSZZ7M23LKIASJHQ3YQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.880497017 (613h23m21.751580062s) [noise=true , address=145.79.227.21:4473 , piece_id=VFSECHPXN7LZVVPX7JBOYYHASVNZZ6CIOZA5SLOA7SEYTQKCZNFQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.880576788 (613h23m21.751659824s) [noise=true , address=145.79.227.22:4511 , piece_id=VHEIWH7UQUPWQKOYZLJXOHYXIRVQTTNITSE5S6PCHEUMPJNTA4PA, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.881771673 (613h23m21.752854708s) [noise=true , address=77.90.31.19:4844 , piece_id=S4GCNZNHKVUOJKXXRJQDP5I6FMRFJAXJKHHF3CZJL2R7FD2PM4HQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.882586725 (613h23m21.75366977s) [noise=true , address=77.90.31.11:4555 , piece_id=HH2B3AGQFXV6VIHSXL5SBEICYIIRRJAWQPLFQKDUE2Y2OJYO6JEQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.882727344 (613h23m21.753810389s) [noise=true , address=145.79.227.37:4565 , piece_id=KLPKMCT3OW3ZWLFNPDLUZJKO5ZYLHGDHLZ6T3MD2QRFRUNHRQYEA, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.883240861 (613h23m21.754323896s) [noise=true , address=145.79.227.41:4085 , piece_id=JNEAUJ5UHTU7QNX3VGUUI262MF5WT63V6YSMVR4ZFBT3MCURB4GA, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.883613911 (613h23m21.754696957s) [noise=true , address=45.91.116.10:4082 , piece_id=CCXTESJT76SQ2UKBAZPJXKRWBWYUYANLJFARBQ3N24QTBY5DUP6A, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.884089247 (613h23m21.755172292s) [noise=true , address=77.90.31.12:4589 , piece_id=KOWJSZ4QJ7I74LJ56DNJIWIEVG2PYDTMR5GV7YGNOVBN6N4E5BQQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.884635176 (613h23m21.755718211s) [noise=true , address=145.79.227.4:4186 , piece_id=A6OISTIMY3RT5JWUG6OV6OTBT235O5AE3H24DLV7KITBXFQZY4ZQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.884975454 (613h23m21.756058489s) [noise=true , address=145.79.227.16:4909 , piece_id=T2NPZNHXE426UCEYBAVIUPTOGZQTHLBHAHMSZCHUFPYU5EOSVOZQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.885934921 (613h23m21.757017956s) [noise=true , address=45.91.116.13:4155 , piece_id=RVOYY7MEK2QR4GQGIG3RZKMFUSNQ5H7BXKNOGQ4BZPFKAJLO6OBA, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.887162489 (613h23m21.758245534s) [noise=true , address=145.79.227.46:4601 , piece_id=JO2LVBIKZH4HXU5HSWIQMJDOT52M6RU4B7AC4AJC4RZFG573RCMQ, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:21.887729949 (613h23m21.758812984s) [noise=true , address=145.79.227.39:4015 , piece_id=YAUFJBFWHOZRAFMMU3JBVDGA2JYM2KLCW63PBV6SQJDGMNDD6P4Q, seg_pos={0 0}, upload=537532]: Uploading piece... Dec 31 04:52:23.625922551 (613h23m23.497005596s) [upload=537532]: Next returned nil segment Dec 31 04:52:23.625937369 (613h23m23.497020404s) [upload=537532]: Waiting for error group managing segments... Dec 31 04:52:23.881727836 (613h23m23.752810871s) [noise=true , address=145.79.227.21:4473 , piece_id=VFSECHPXN7LZVVPX7JBOYYHASVNZZ6CIOZA5SLOA7SEYTQKCZNFQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.883922876 (613h23m23.755005921s) [noise=true , address=145.79.227.9:4366 , piece_id=OQ76IWMTLCX6LTUJMR5CDMHC3WOS742EPOSZZ7M23LKIASJHQ3YQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.886029607 (613h23m23.757112642s) [noise=true , address=145.79.227.16:4909 , piece_id=T2NPZNHXE426UCEYBAVIUPTOGZQTHLBHAHMSZCHUFPYU5EOSVOZQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.890565283 (613h23m23.761648328s) [noise=true , address=145.79.227.4:4186 , piece_id=A6OISTIMY3RT5JWUG6OV6OTBT235O5AE3H24DLV7KITBXFQZY4ZQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.893963995 (613h23m23.76504704s) [noise=true , address=145.79.227.39:4015 , piece_id=YAUFJBFWHOZRAFMMU3JBVDGA2JYM2KLCW63PBV6SQJDGMNDD6P4Q, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.894813392 (613h23m23.765896427s) [noise=true , address=145.79.227.41:4085 , piece_id=JNEAUJ5UHTU7QNX3VGUUI262MF5WT63V6YSMVR4ZFBT3MCURB4GA, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.895110198 (613h23m23.766193233s) [noise=true , address=145.79.227.10:4403 , piece_id=PPIXOQODOH6NNBPKU6PNOB764ELLAC7QFALF7QSF5C4TV2532WWQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.937499191 (613h23m23.808582236s) [noise=true , address=145.79.227.13:4803 , piece_id=EQBPPBHGTHIJXLQUH45DL2M3DXVMZU736F6YREYGXAYMLJYDQFSA, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.940935795 (613h23m23.812018841s) [noise=true , address=145.79.227.46:4601 , piece_id=JO2LVBIKZH4HXU5HSWIQMJDOT52M6RU4B7AC4AJC4RZFG573RCMQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:23.963207784 (613h23m23.834290819s) [noise=true , address=145.79.227.37:4565 , piece_id=KLPKMCT3OW3ZWLFNPDLUZJKO5ZYLHGDHLZ6T3MD2QRFRUNHRQYEA, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:24.041218496 (613h23m23.912301531s) [noise=true , address=145.79.227.22:4511 , piece_id=VHEIWH7UQUPWQKOYZLJXOHYXIRVQTTNITSE5S6PCHEUMPJNTA4PA, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:24.158777742 (613h23m24.029860787s) [noise=true , address=77.90.31.12:4589 , piece_id=KOWJSZ4QJ7I74LJ56DNJIWIEVG2PYDTMR5GV7YGNOVBN6N4E5BQQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:24.187116573 (613h23m24.058199628s) [noise=true , address=77.90.31.19:4844 , piece_id=S4GCNZNHKVUOJKXXRJQDP5I6FMRFJAXJKHHF3CZJL2R7FD2PM4HQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: <nil> Dec 31 04:52:24.187143584 (613h23m24.058226619s) [seg_pos={0 0}, upload=537532]: Segment reached optimal threshold of 13 pieces. Dec 31 04:52:24.187566188 (613h23m24.058649223s) [noise=true , address=45.91.116.13:4155 , piece_id=RVOYY7MEK2QR4GQGIG3RZKMFUSNQ5H7BXKNOGQ4BZPFKAJLO6OBA, seg_pos={0 0}, upload=537532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 04:52:24.187611756 (613h23m24.058694791s) [noise=true , address=45.91.116.13:4155 , piece_id=RVOYY7MEK2QR4GQGIG3RZKMFUSNQ5H7BXKNOGQ4BZPFKAJLO6OBA, seg_pos={0 0}, upload=537532]: Long tail cancellation due to optimal threshold Dec 31 04:52:24.187792911 (613h23m24.058875966s) [noise=true , address=77.90.31.11:4555 , piece_id=HH2B3AGQFXV6VIHSXL5SBEICYIIRRJAWQPLFQKDUE2Y2OJYO6JEQ, seg_pos={0 0}, upload=537532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 31 04:52:24.187806506 (613h23m24.058889541s) [noise=true , address=77.90.31.11:4555 , piece_id=HH2B3AGQFXV6VIHSXL5SBEICYIIRRJAWQPLFQKDUE2Y2OJYO6JEQ, seg_pos={0 0}, upload=537532]: Long tail cancellation due to optimal threshold Dec 31 04:52:24.188023861 (613h23m24.059106906s) [noise=true , address=45.91.116.10:4082 , piece_id=CCXTESJT76SQ2UKBAZPJXKRWBWYUYANLJFARBQ3N24QTBY5DUP6A, seg_pos={0 0}, upload=537532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 31 04:52:24.188049659 (613h23m24.059132694s) [noise=true , address=45.91.116.10:4082 , piece_id=CCXTESJT76SQ2UKBAZPJXKRWBWYUYANLJFARBQ3N24QTBY5DUP6A, seg_pos={0 0}, upload=537532]: Long tail cancellation due to optimal threshold Dec 31 04:52:24.188072353 (613h23m24.059155388s) [seg_pos={0 0}, upload=537532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:52:24.188170530 (613h23m24.059253575s) [upload=537532]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:52:24.188176060 (613h23m24.059259095s) [upload=537532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:52:24.475797317 (613h23m24.346880362s) [upload=537532]: Done uploading segments... Dec 31 04:52:24.475819960 (613h23m24.346903005s) [upload=537532]: Upload finished. err: <nil> Duration: 30.002980737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X...X......X...X..X.....X...........X..............X.....] duration: 4.634407867s trace: https://jaeger.dev.storj.io/trace/7f45acefcc83d80 FAIL (2025-12-31 04:52:49 UTC) orbiter warning: context deadline exceeded Duration: 4.634437682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....X...X......X...X..X.....X...........X..............X....] duration: 3.514870024s trace: https://jaeger.dev.storj.io/trace/7f45acefcc83d80 FAIL (2025-12-31 04:52:31 UTC) orbiter warning: context deadline exceeded Duration: 3.514889471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.330059997s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:52:28 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.330089333s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX........X.....X.............X............................] duration: 15.443978849s trace: https://jaeger.dev.storj.io/trace/5943c4171683f71a FAIL (2025-12-31 04:52:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments... Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces. Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments... Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil> Duration: 15.444007655s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....X...X......X...X..X.....X...........X..............X...] duration: 3.582799872s trace: https://jaeger.dev.storj.io/trace/7f45acefcc83d80 FAIL (2025-12-31 04:52:12 UTC) orbiter warning: context deadline exceeded Duration: 3.582829339s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.078362809s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:51:59 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.078380131s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X........XX.X...X...X..X...X.......X.......X.........XX...X.] duration: 27.217624978s trace: https://jaeger.dev.storj.io/trace/3390bb4bd838953e FAIL (2025-12-31 04:51:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: QSUZJPE7FE6H3EXNYSHLQKVUUWWK5AJ4U3R2FQBUAMLPQO7DXCUQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 3U7WKLGLKUNRKN62TWI6DV5YJYWPGAHAJ4D4VC7XZ6W6HTL2VS7Q) context deadline exceeded ======LOGS====== 31 04:50:53.828646884 (613h21m53.699729929s) [upload=537513]: Starting upload Dec 31 04:50:53.828767074 (613h21m53.699850119s) [upload=537513]: Done starting upload Dec 31 04:50:53.828946736 (613h21m53.700029791s) [upload=537513]: Uploading segments... Dec 31 04:50:53.829013583 (613h21m53.700096618s) [upload=537513]: Got next segment. Inline: false Dec 31 04:50:53.829054551 (613h21m53.700137596s) [upload=537513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:50:53.829060622 (613h21m53.700143657s) [upload=537513]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:50:54.078130342 (613h21m53.949213387s) [seg_pos={0 0}, upload=537513]: Begin upload segment... Dec 31 04:50:54.078181218 (613h21m53.949264263s) [seg_pos={0 0}, upload=537513]: Done begin upload segment. Dec 31 04:50:54.078214522 (613h21m53.949297567s) [noise=true , address=77.90.31.12:4589 , piece_id=NWM326S2LRLB2QE2JBWYRUEBDRU3AD3FBHJ3L5D2TOAS35O7ABGQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.079261696 (613h21m53.950344731s) [noise=true , address=145.79.227.47:4265 , piece_id=H654O7JG6TBYZ6GVMWXLZZHLJD57GMAOPQDRN64HG2QZD44TRH2A, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.079788809 (613h21m53.950871845s) [noise=true , address=77.90.31.11:4556 , piece_id=3U7WKLGLKUNRKN62TWI6DV5YJYWPGAHAJ4D4VC7XZ6W6HTL2VS7Q, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.080526956 (613h21m53.951610001s) [noise=true , address=77.90.31.19:4841 , piece_id=UUBABUAJRYJHETAASRNSQETXXFIXYCWN3LWMJKMR4A7ZL3VHCNVQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.080913331 (613h21m53.951996366s) [noise=true , address=45.91.116.8:4048 , piece_id=LOIICTVRUFC6KUCJBRGAKYHNSOXKV3H7XMWMG54DVW6Z7SXIOF5A, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.081222480 (613h21m53.952305515s) [noise=true , address=145.79.227.41:4086 , piece_id=PE7MX6WYHB3OFUR5P37HZYO3RGGL23NQ742MEQXJTSTXSM7E6ITQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.081553320 (613h21m53.952636355s) [noise=true , address=145.79.227.42:4122 , piece_id=XRWEHJ4FXOKIQIHXQIAM6KVZTY5HQEPDZC6UFJAG3EJXH75LPQPQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.082043653 (613h21m53.953126698s) [noise=true , address=145.79.227.12:4767 , piece_id=4TMQ4FQ5LUWBSRJ22MY773TFCQP67M2PFXLXDICZFV2M7WZ6QGZA, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.082557552 (613h21m53.953640587s) [noise=true , address=145.79.227.20:4438 , piece_id=6ENX777S6KUDNMNHFP5JLRKDJOIRUO2654M5PK5GLYSSVJZ2IFSA, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.084249373 (613h21m53.955332418s) [noise=true , address=45.91.116.4:4937 , piece_id=2TTU7WOJXUVP2ZDCD7LT6VGL542PZ54BYEM2NIOOM22UYG5EN4XQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.086089898 (613h21m53.957172933s) [noise=true , address=45.91.116.11:4118 , piece_id=E2OFNHGHL6AZNEOCG34RW74JERNNER4A2RU3FVNNTQRJ2YASBDRQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.087967703 (613h21m53.959050748s) [noise=true , address=145.79.227.15:4875 , piece_id=VL5JOQ7QDK6USTFXVAIMZDXOYZIS7T3ZH6CE64ZZ4KJHV22ITETQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.092246341 (613h21m53.963329396s) [noise=true , address=145.79.227.45:4229 , piece_id=TP7A6HW6AURJQ5VNDT3C5UIRG6JHVF4PC36G5ANDH4FLVIAXKTYA, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.092642715 (613h21m53.96372576s) [noise=true , address=77.90.31.15:4700 , piece_id=QSUZJPE7FE6H3EXNYSHLQKVUUWWK5AJ4U3R2FQBUAMLPQO7DXCUQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.097130771 (613h21m53.968213816s) [noise=true , address=145.79.227.10:4403 , piece_id=3P6TKNVOI6TETE46QEWA6YAB3LKNELTU5E2WCTU2NNTR35RWAPPA, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.100230053 (613h21m53.971313088s) [noise=true , address=145.79.227.2:4113 , piece_id=LEQXHTZMBZ3U64OVQP5QPE4LMTZCGN6ZOIEQUANGELBXNSK3WD3Q, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:55.808988630 (613h21m55.680071675s) [upload=537513]: Next returned nil segment Dec 31 04:50:55.809011915 (613h21m55.68009496s) [upload=537513]: Waiting for error group managing segments... Dec 31 04:50:56.087583990 (613h21m55.958667025s) [noise=true , address=145.79.227.10:4403 , piece_id=3P6TKNVOI6TETE46QEWA6YAB3LKNELTU5E2WCTU2NNTR35RWAPPA, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.090611475 (613h21m55.96169451s) [noise=true , address=145.79.227.41:4086 , piece_id=PE7MX6WYHB3OFUR5P37HZYO3RGGL23NQ742MEQXJTSTXSM7E6ITQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.090866851 (613h21m55.961949896s) [noise=true , address=145.79.227.45:4229 , piece_id=TP7A6HW6AURJQ5VNDT3C5UIRG6JHVF4PC36G5ANDH4FLVIAXKTYA, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.097040026 (613h21m55.968123061s) [noise=true , address=145.79.227.15:4875 , piece_id=VL5JOQ7QDK6USTFXVAIMZDXOYZIS7T3ZH6CE64ZZ4KJHV22ITETQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.103553007 (613h21m55.974636042s) [noise=true , address=145.79.227.20:4438 , piece_id=6ENX777S6KUDNMNHFP5JLRKDJOIRUO2654M5PK5GLYSSVJZ2IFSA, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.104278849 (613h21m55.975361884s) [noise=true , address=145.79.227.42:4122 , piece_id=XRWEHJ4FXOKIQIHXQIAM6KVZTY5HQEPDZC6UFJAG3EJXH75LPQPQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.104509759 (613h21m55.975592804s) [noise=true , address=145.79.227.2:4113 , piece_id=LEQXHTZMBZ3U64OVQP5QPE4LMTZCGN6ZOIEQUANGELBXNSK3WD3Q, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.111279229 (613h21m55.982362264s) [noise=true , address=145.79.227.47:4265 , piece_id=H654O7JG6TBYZ6GVMWXLZZHLJD57GMAOPQDRN64HG2QZD44TRH2A, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.116965316 (613h21m55.988048351s) [noise=true , address=145.79.227.12:4767 , piece_id=4TMQ4FQ5LUWBSRJ22MY773TFCQP67M2PFXLXDICZFV2M7WZ6QGZA, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.350650692 (613h21m56.221733737s) [noise=true , address=77.90.31.12:4589 , piece_id=NWM326S2LRLB2QE2JBWYRUEBDRU3AD3FBHJ3L5D2TOAS35O7ABGQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.360684308 (613h21m56.231767373s) [noise=true , address=77.90.31.11:4556 , piece_id=3U7WKLGLKUNRKN62TWI6DV5YJYWPGAHAJ4D4VC7XZ6W6HTL2VS7Q, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.372990530 (613h21m56.244073565s) [noise=true , address=77.90.31.15:4700 , piece_id=QSUZJPE7FE6H3EXNYSHLQKVUUWWK5AJ4U3R2FQBUAMLPQO7DXCUQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.406353237 (613h21m56.277436282s) [noise=true , address=45.91.116.4:4937 , piece_id=2TTU7WOJXUVP2ZDCD7LT6VGL542PZ54BYEM2NIOOM22UYG5EN4XQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.406378024 (613h21m56.277461059s) [seg_pos={0 0}, upload=537513]: Segment reached optimal threshold of 13 pieces. Dec 31 04:50:56.406628201 (613h21m56.277711266s) [noise=true , address=45.91.116.8:4048 , piece_id=LOIICTVRUFC6KUCJBRGAKYHNSOXKV3H7XMWMG54DVW6Z7SXIOF5A, seg_pos={0 0}, upload=537513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 04:50:56.406638210 (613h21m56.277721245s) [noise=true , address=45.91.116.8:4048 , piece_id=LOIICTVRUFC6KUCJBRGAKYHNSOXKV3H7XMWMG54DVW6Z7SXIOF5A, seg_pos={0 0}, upload=537513]: Long tail cancellation due to optimal threshold Dec 31 04:50:56.406687784 (613h21m56.277770819s) [noise=true , address=77.90.31.19:4841 , piece_id=UUBABUAJRYJHETAASRNSQETXXFIXYCWN3LWMJKMR4A7ZL3VHCNVQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:57876->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:57876->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57876->77.90.31.19:4841: use of closed network connection Dec 31 04:50:56.406707722 (613h21m56.277790757s) [noise=true , address=77.90.31.19:4841 , piece_id=UUBABUAJRYJHETAASRNSQETXXFIXYCWN3LWMJKMR4A7ZL3VHCNVQ, seg_pos={0 0}, upload=537513]: Long tail cancellation due to optimal threshold Dec 31 04:50:56.406956476 (613h21m56.278039511s) [noise=true , address=45.91.116.11:4118 , piece_id=E2OFNHGHL6AZNEOCG34RW74JERNNER4A2RU3FVNNTQRJ2YASBDRQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 04:50:56.406964751 (613h21m56.278047786s) [noise=true , address=45.91.116.11:4118 , piece_id=E2OFNHGHL6AZNEOCG34RW74JERNNER4A2RU3FVNNTQRJ2YASBDRQ, seg_pos={0 0}, upload=537513]: Long tail cancellation due to optimal threshold Dec 31 04:50:56.406982445 (613h21m56.27806548s) [seg_pos={0 0}, upload=537513]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:50:56.407055434 (613h21m56.278138469s) [upload=537513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:50:56.407060914 (613h21m56.278143949s) [upload=537513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:50:56.711678544 (613h21m56.582761579s) [upload=537513]: Done uploading segments... Dec 31 04:50:56.711698191 (613h21m56.582781236s) [upload=537513]: Upload finished. err: <nil> Duration: 27.217654964s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 7.589385405s trace: https://jaeger.dev.storj.io/trace/1080f2dcf8b139c9 FAIL (2025-12-31 04:51:55 UTC) orbiter warning: context deadline exceeded Duration: 7.589421575s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX........X.....X.............X...........................] duration: 15.414735882s trace: https://jaeger.dev.storj.io/trace/5943c4171683f71a FAIL (2025-12-31 04:51:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments... Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces. Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments... Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil> Duration: 15.414767351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.074240501s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:51:32 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.074269174s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X........XX.X...X...X..X...X.......X.......X.........XX...X] duration: 30.004812885s trace: https://jaeger.dev.storj.io/trace/3390bb4bd838953e FAIL (2025-12-31 04:51:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: QSUZJPE7FE6H3EXNYSHLQKVUUWWK5AJ4U3R2FQBUAMLPQO7DXCUQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 3U7WKLGLKUNRKN62TWI6DV5YJYWPGAHAJ4D4VC7XZ6W6HTL2VS7Q) context deadline exceeded ======LOGS====== 31 04:50:53.828646884 (613h21m53.699729929s) [upload=537513]: Starting upload Dec 31 04:50:53.828767074 (613h21m53.699850119s) [upload=537513]: Done starting upload Dec 31 04:50:53.828946736 (613h21m53.700029791s) [upload=537513]: Uploading segments... Dec 31 04:50:53.829013583 (613h21m53.700096618s) [upload=537513]: Got next segment. Inline: false Dec 31 04:50:53.829054551 (613h21m53.700137596s) [upload=537513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:50:53.829060622 (613h21m53.700143657s) [upload=537513]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:50:54.078130342 (613h21m53.949213387s) [seg_pos={0 0}, upload=537513]: Begin upload segment... Dec 31 04:50:54.078181218 (613h21m53.949264263s) [seg_pos={0 0}, upload=537513]: Done begin upload segment. Dec 31 04:50:54.078214522 (613h21m53.949297567s) [noise=true , address=77.90.31.12:4589 , piece_id=NWM326S2LRLB2QE2JBWYRUEBDRU3AD3FBHJ3L5D2TOAS35O7ABGQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.079261696 (613h21m53.950344731s) [noise=true , address=145.79.227.47:4265 , piece_id=H654O7JG6TBYZ6GVMWXLZZHLJD57GMAOPQDRN64HG2QZD44TRH2A, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.079788809 (613h21m53.950871845s) [noise=true , address=77.90.31.11:4556 , piece_id=3U7WKLGLKUNRKN62TWI6DV5YJYWPGAHAJ4D4VC7XZ6W6HTL2VS7Q, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.080526956 (613h21m53.951610001s) [noise=true , address=77.90.31.19:4841 , piece_id=UUBABUAJRYJHETAASRNSQETXXFIXYCWN3LWMJKMR4A7ZL3VHCNVQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.080913331 (613h21m53.951996366s) [noise=true , address=45.91.116.8:4048 , piece_id=LOIICTVRUFC6KUCJBRGAKYHNSOXKV3H7XMWMG54DVW6Z7SXIOF5A, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.081222480 (613h21m53.952305515s) [noise=true , address=145.79.227.41:4086 , piece_id=PE7MX6WYHB3OFUR5P37HZYO3RGGL23NQ742MEQXJTSTXSM7E6ITQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.081553320 (613h21m53.952636355s) [noise=true , address=145.79.227.42:4122 , piece_id=XRWEHJ4FXOKIQIHXQIAM6KVZTY5HQEPDZC6UFJAG3EJXH75LPQPQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.082043653 (613h21m53.953126698s) [noise=true , address=145.79.227.12:4767 , piece_id=4TMQ4FQ5LUWBSRJ22MY773TFCQP67M2PFXLXDICZFV2M7WZ6QGZA, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.082557552 (613h21m53.953640587s) [noise=true , address=145.79.227.20:4438 , piece_id=6ENX777S6KUDNMNHFP5JLRKDJOIRUO2654M5PK5GLYSSVJZ2IFSA, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.084249373 (613h21m53.955332418s) [noise=true , address=45.91.116.4:4937 , piece_id=2TTU7WOJXUVP2ZDCD7LT6VGL542PZ54BYEM2NIOOM22UYG5EN4XQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.086089898 (613h21m53.957172933s) [noise=true , address=45.91.116.11:4118 , piece_id=E2OFNHGHL6AZNEOCG34RW74JERNNER4A2RU3FVNNTQRJ2YASBDRQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.087967703 (613h21m53.959050748s) [noise=true , address=145.79.227.15:4875 , piece_id=VL5JOQ7QDK6USTFXVAIMZDXOYZIS7T3ZH6CE64ZZ4KJHV22ITETQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.092246341 (613h21m53.963329396s) [noise=true , address=145.79.227.45:4229 , piece_id=TP7A6HW6AURJQ5VNDT3C5UIRG6JHVF4PC36G5ANDH4FLVIAXKTYA, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.092642715 (613h21m53.96372576s) [noise=true , address=77.90.31.15:4700 , piece_id=QSUZJPE7FE6H3EXNYSHLQKVUUWWK5AJ4U3R2FQBUAMLPQO7DXCUQ, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.097130771 (613h21m53.968213816s) [noise=true , address=145.79.227.10:4403 , piece_id=3P6TKNVOI6TETE46QEWA6YAB3LKNELTU5E2WCTU2NNTR35RWAPPA, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:54.100230053 (613h21m53.971313088s) [noise=true , address=145.79.227.2:4113 , piece_id=LEQXHTZMBZ3U64OVQP5QPE4LMTZCGN6ZOIEQUANGELBXNSK3WD3Q, seg_pos={0 0}, upload=537513]: Uploading piece... Dec 31 04:50:55.808988630 (613h21m55.680071675s) [upload=537513]: Next returned nil segment Dec 31 04:50:55.809011915 (613h21m55.68009496s) [upload=537513]: Waiting for error group managing segments... Dec 31 04:50:56.087583990 (613h21m55.958667025s) [noise=true , address=145.79.227.10:4403 , piece_id=3P6TKNVOI6TETE46QEWA6YAB3LKNELTU5E2WCTU2NNTR35RWAPPA, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.090611475 (613h21m55.96169451s) [noise=true , address=145.79.227.41:4086 , piece_id=PE7MX6WYHB3OFUR5P37HZYO3RGGL23NQ742MEQXJTSTXSM7E6ITQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.090866851 (613h21m55.961949896s) [noise=true , address=145.79.227.45:4229 , piece_id=TP7A6HW6AURJQ5VNDT3C5UIRG6JHVF4PC36G5ANDH4FLVIAXKTYA, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.097040026 (613h21m55.968123061s) [noise=true , address=145.79.227.15:4875 , piece_id=VL5JOQ7QDK6USTFXVAIMZDXOYZIS7T3ZH6CE64ZZ4KJHV22ITETQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.103553007 (613h21m55.974636042s) [noise=true , address=145.79.227.20:4438 , piece_id=6ENX777S6KUDNMNHFP5JLRKDJOIRUO2654M5PK5GLYSSVJZ2IFSA, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.104278849 (613h21m55.975361884s) [noise=true , address=145.79.227.42:4122 , piece_id=XRWEHJ4FXOKIQIHXQIAM6KVZTY5HQEPDZC6UFJAG3EJXH75LPQPQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.104509759 (613h21m55.975592804s) [noise=true , address=145.79.227.2:4113 , piece_id=LEQXHTZMBZ3U64OVQP5QPE4LMTZCGN6ZOIEQUANGELBXNSK3WD3Q, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.111279229 (613h21m55.982362264s) [noise=true , address=145.79.227.47:4265 , piece_id=H654O7JG6TBYZ6GVMWXLZZHLJD57GMAOPQDRN64HG2QZD44TRH2A, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.116965316 (613h21m55.988048351s) [noise=true , address=145.79.227.12:4767 , piece_id=4TMQ4FQ5LUWBSRJ22MY773TFCQP67M2PFXLXDICZFV2M7WZ6QGZA, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.350650692 (613h21m56.221733737s) [noise=true , address=77.90.31.12:4589 , piece_id=NWM326S2LRLB2QE2JBWYRUEBDRU3AD3FBHJ3L5D2TOAS35O7ABGQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.360684308 (613h21m56.231767373s) [noise=true , address=77.90.31.11:4556 , piece_id=3U7WKLGLKUNRKN62TWI6DV5YJYWPGAHAJ4D4VC7XZ6W6HTL2VS7Q, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.372990530 (613h21m56.244073565s) [noise=true , address=77.90.31.15:4700 , piece_id=QSUZJPE7FE6H3EXNYSHLQKVUUWWK5AJ4U3R2FQBUAMLPQO7DXCUQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.406353237 (613h21m56.277436282s) [noise=true , address=45.91.116.4:4937 , piece_id=2TTU7WOJXUVP2ZDCD7LT6VGL542PZ54BYEM2NIOOM22UYG5EN4XQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: <nil> Dec 31 04:50:56.406378024 (613h21m56.277461059s) [seg_pos={0 0}, upload=537513]: Segment reached optimal threshold of 13 pieces. Dec 31 04:50:56.406628201 (613h21m56.277711266s) [noise=true , address=45.91.116.8:4048 , piece_id=LOIICTVRUFC6KUCJBRGAKYHNSOXKV3H7XMWMG54DVW6Z7SXIOF5A, seg_pos={0 0}, upload=537513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 31 04:50:56.406638210 (613h21m56.277721245s) [noise=true , address=45.91.116.8:4048 , piece_id=LOIICTVRUFC6KUCJBRGAKYHNSOXKV3H7XMWMG54DVW6Z7SXIOF5A, seg_pos={0 0}, upload=537513]: Long tail cancellation due to optimal threshold Dec 31 04:50:56.406687784 (613h21m56.277770819s) [noise=true , address=77.90.31.19:4841 , piece_id=UUBABUAJRYJHETAASRNSQETXXFIXYCWN3LWMJKMR4A7ZL3VHCNVQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:57876->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:57876->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57876->77.90.31.19:4841: use of closed network connection Dec 31 04:50:56.406707722 (613h21m56.277790757s) [noise=true , address=77.90.31.19:4841 , piece_id=UUBABUAJRYJHETAASRNSQETXXFIXYCWN3LWMJKMR4A7ZL3VHCNVQ, seg_pos={0 0}, upload=537513]: Long tail cancellation due to optimal threshold Dec 31 04:50:56.406956476 (613h21m56.278039511s) [noise=true , address=45.91.116.11:4118 , piece_id=E2OFNHGHL6AZNEOCG34RW74JERNNER4A2RU3FVNNTQRJ2YASBDRQ, seg_pos={0 0}, upload=537513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 04:50:56.406964751 (613h21m56.278047786s) [noise=true , address=45.91.116.11:4118 , piece_id=E2OFNHGHL6AZNEOCG34RW74JERNNER4A2RU3FVNNTQRJ2YASBDRQ, seg_pos={0 0}, upload=537513]: Long tail cancellation due to optimal threshold Dec 31 04:50:56.406982445 (613h21m56.27806548s) [seg_pos={0 0}, upload=537513]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:50:56.407055434 (613h21m56.278138469s) [upload=537513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:50:56.407060914 (613h21m56.278143949s) [upload=537513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:50:56.711678544 (613h21m56.582761579s) [upload=537513]: Done uploading segments... Dec 31 04:50:56.711698191 (613h21m56.582781236s) [upload=537513]: Upload finished. err: <nil> Duration: 30.004840638s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......X...X......X...X..X.....X...........X..............X..] duration: 3.711603756s trace: https://jaeger.dev.storj.io/trace/7f45acefcc83d80 FAIL (2025-12-31 04:51:23 UTC) orbiter warning: context deadline exceeded Duration: 3.71163271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX........X.....X.............X..........................] duration: 26.183189s trace: https://jaeger.dev.storj.io/trace/5943c4171683f71a FAIL (2025-12-31 04:51:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments... Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces. Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments... Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil> Duration: 26.183214709s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 16.540681515s trace: https://jaeger.dev.storj.io/trace/1080f2dcf8b139c9 FAIL (2025-12-31 04:51:03 UTC) orbiter warning: context deadline exceeded Duration: 16.540706204s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.881362188s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:50:48 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.88138369s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......X...X......X...X..X.....X...........X..............X.] duration: 3.614997579s trace: https://jaeger.dev.storj.io/trace/7f45acefcc83d80 FAIL (2025-12-31 04:50:48 UTC) orbiter warning: context deadline exceeded Duration: 3.615014722s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 15.475743723s trace: https://jaeger.dev.storj.io/trace/1080f2dcf8b139c9 FAIL (2025-12-31 04:50:44 UTC) orbiter warning: context deadline exceeded Duration: 15.475760113s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX........X.....X.............X.........................] duration: 44.796577446s trace: https://jaeger.dev.storj.io/trace/5943c4171683f71a FAIL (2025-12-31 04:50:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments... Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces. Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments... Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil> Duration: 44.796593505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X........XX.X...X...X..X...X.......X.......X.........XX...] duration: 5.825767472s trace: https://jaeger.dev.storj.io/trace/7e73fd1b2a67fbfe FAIL (2025-12-31 04:50:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA) context deadline exceeded ======LOGS====== Dec 31 04:48:29.244427553 (613h19m29.115510598s) [upload=537472]: Starting upload Dec 31 04:48:29.244481615 (613h19m29.115564651s) [upload=537472]: Done starting upload Dec 31 04:48:29.244654815 (613h19m29.11573785s) [upload=537472]: Uploading segments... Dec 31 04:48:29.244692277 (613h19m29.115775332s) [upload=537472]: Got next segment. Inline: false Dec 31 04:48:29.244719358 (613h19m29.115802403s) [upload=537472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:48:29.244723957 (613h19m29.115806992s) [upload=537472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:48:29.492669225 (613h19m29.36375226s) [seg_pos={0 0}, upload=537472]: Begin upload segment... Dec 31 04:48:29.492717817 (613h19m29.363800852s) [seg_pos={0 0}, upload=537472]: Done begin upload segment. Dec 31 04:48:29.492750419 (613h19m29.363833454s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.492807709 (613h19m29.363890754s) [noise=true , address=45.91.116.10:4084 , piece_id=Z7NS7OSJ4ZVKMJA24OPYZJ7LVYY4URBXKON7B46JYWCTCFX7AGUA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.493560702 (613h19m29.364643737s) [noise=true , address=145.79.227.46:4603 , piece_id=TH74SWTPYY2RS5HCYD6HVZ7AKT7PECP54QW2O2QBPOGE6KBNZZ3Q, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.493939974 (613h19m29.365023009s) [noise=true , address=45.91.116.7:4010 , piece_id=WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494242320 (613h19m29.365325355s) [noise=true , address=145.79.227.10:4402 , piece_id=ZRVS7X3J5F4SOMWLXMK43YG3B53CY5H2TSBNWD2GUGIUVEBVOIVA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494532452 (613h19m29.365615487s) [noise=true , address=145.79.227.8:4331 , piece_id=VYSOLGYMWJFNNKDMCIO2H4HPP25XSPQR2A2U3DAMW32COJUV56DA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494831212 (613h19m29.365914247s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.495135831 (613h19m29.366218866s) [noise=true , address=145.79.227.4:4186 , piece_id=HLLYRK34RYCEHAEJYB2U4VONMZDQNOUPGV474QWNFJT6JXJXB2QA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.495582452 (613h19m29.366665487s) [noise=true , address=45.91.116.11:4117 , piece_id=QEMAR37LSEBTIWVJ55BE45B72F4EABQD7L4LBZFRUSIPC24X5KMA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.496876726 (613h19m29.367959771s) [noise=true , address=145.79.227.48:4302 , piece_id=HBOPVTDU6TIZHKPTH6TPWMGNNQTBWSR3KVTXCFDHHV2DB2EMEZPQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.497315261 (613h19m29.368398296s) [noise=true , address=77.90.31.16:4735 , piece_id=64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.498610377 (613h19m29.369693412s) [noise=true , address=145.79.227.38:4978 , piece_id=UNIKEWLH6FUFTUNBAV5EYKFMD2BSFEFQM5SCI7T5ATG64OT6UPWA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.500086187 (613h19m29.371169232s) [noise=true , address=145.79.227.20:4437 , piece_id=K72VWZ5ULZTUXF7JYIQP7PHAL66QWYH666DWKRJNEGIHQFEXLGCQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.501835528 (613h19m29.372918563s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.503464039 (613h19m29.374547084s) [noise=true , address=145.79.227.42:4123 , piece_id=UA5ORVLFUIJMB7HEBJLQWITPNJJ5CVRT7DIWX6NR4PFHMVJAEPGQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.506372438 (613h19m29.377455473s) [noise=true , address=145.79.227.18:4007 , piece_id=5NBDLAJ7XXF2AQVMNLUJ5JLTKIPEQFPEDACN5ZTWEMQSHCR7G4UA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:31.208953591 (613h19m31.080036626s) [upload=537472]: Next returned nil segment Dec 31 04:48:31.208969060 (613h19m31.080052105s) [upload=537472]: Waiting for error group managing segments... Dec 31 04:48:31.479989536 (613h19m31.351072581s) [noise=true , address=145.79.227.46:4603 , piece_id=TH74SWTPYY2RS5HCYD6HVZ7AKT7PECP54QW2O2QBPOGE6KBNZZ3Q, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.497388966 (613h19m31.368472012s) [noise=true , address=145.79.227.4:4186 , piece_id=HLLYRK34RYCEHAEJYB2U4VONMZDQNOUPGV474QWNFJT6JXJXB2QA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.499652016 (613h19m31.370735061s) [noise=true , address=145.79.227.8:4331 , piece_id=VYSOLGYMWJFNNKDMCIO2H4HPP25XSPQR2A2U3DAMW32COJUV56DA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.504212910 (613h19m31.375295945s) [noise=true , address=145.79.227.10:4402 , piece_id=ZRVS7X3J5F4SOMWLXMK43YG3B53CY5H2TSBNWD2GUGIUVEBVOIVA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.520135910 (613h19m31.391218955s) [noise=true , address=145.79.227.20:4437 , piece_id=K72VWZ5ULZTUXF7JYIQP7PHAL66QWYH666DWKRJNEGIHQFEXLGCQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.524280291 (613h19m31.395363336s) [noise=true , address=145.79.227.18:4007 , piece_id=5NBDLAJ7XXF2AQVMNLUJ5JLTKIPEQFPEDACN5ZTWEMQSHCR7G4UA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.526111558 (613h19m31.397194593s) [noise=true , address=145.79.227.48:4302 , piece_id=HBOPVTDU6TIZHKPTH6TPWMGNNQTBWSR3KVTXCFDHHV2DB2EMEZPQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.582629986 (613h19m31.453713031s) [noise=true , address=145.79.227.38:4978 , piece_id=UNIKEWLH6FUFTUNBAV5EYKFMD2BSFEFQM5SCI7T5ATG64OT6UPWA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.593131392 (613h19m31.464214437s) [noise=true , address=145.79.227.42:4123 , piece_id=UA5ORVLFUIJMB7HEBJLQWITPNJJ5CVRT7DIWX6NR4PFHMVJAEPGQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.751417148 (613h19m31.622500183s) [noise=true , address=77.90.31.16:4735 , piece_id=64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.819681973 (613h19m31.690765018s) [noise=true , address=45.91.116.7:4010 , piece_id=WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.854844226 (613h19m31.725927261s) [noise=true , address=45.91.116.11:4117 , piece_id=QEMAR37LSEBTIWVJ55BE45B72F4EABQD7L4LBZFRUSIPC24X5KMA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.893142410 (613h19m31.764225495s) [noise=true , address=45.91.116.10:4084 , piece_id=Z7NS7OSJ4ZVKMJA24OPYZJ7LVYY4URBXKON7B46JYWCTCFX7AGUA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.893168720 (613h19m31.764251775s) [seg_pos={0 0}, upload=537472]: Segment reached optimal threshold of 13 pieces. Dec 31 04:48:31.893387546 (613h19m31.764470643s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893397095 (613h19m31.76448015s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893588169 (613h19m31.764671204s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893622604 (613h19m31.764705649s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893764454 (613h19m31.764847499s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893775706 (613h19m31.764858751s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893830881 (613h19m31.764913916s) [seg_pos={0 0}, upload=537472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:48:31.893918268 (613h19m31.765001323s) [upload=537472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:48:31.893924299 (613h19m31.765007334s) [upload=537472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:48:32.194870107 (613h19m32.065953142s) [upload=537472]: Done uploading segments... Dec 31 04:48:32.194894533 (613h19m32.065977569s) [upload=537472]: Upload finished. err: <nil> Duration: 5.825800145s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.034724277s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:50:32 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 6.034754747s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.324289495s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:50:12 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.324316888s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X........XX.X...X...X..X...X.......X.......X.........XX..] duration: 5.821551862s trace: https://jaeger.dev.storj.io/trace/7e73fd1b2a67fbfe FAIL (2025-12-31 04:49:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA) context deadline exceeded ======LOGS====== Dec 31 04:48:29.244427553 (613h19m29.115510598s) [upload=537472]: Starting upload Dec 31 04:48:29.244481615 (613h19m29.115564651s) [upload=537472]: Done starting upload Dec 31 04:48:29.244654815 (613h19m29.11573785s) [upload=537472]: Uploading segments... Dec 31 04:48:29.244692277 (613h19m29.115775332s) [upload=537472]: Got next segment. Inline: false Dec 31 04:48:29.244719358 (613h19m29.115802403s) [upload=537472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:48:29.244723957 (613h19m29.115806992s) [upload=537472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:48:29.492669225 (613h19m29.36375226s) [seg_pos={0 0}, upload=537472]: Begin upload segment... Dec 31 04:48:29.492717817 (613h19m29.363800852s) [seg_pos={0 0}, upload=537472]: Done begin upload segment. Dec 31 04:48:29.492750419 (613h19m29.363833454s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.492807709 (613h19m29.363890754s) [noise=true , address=45.91.116.10:4084 , piece_id=Z7NS7OSJ4ZVKMJA24OPYZJ7LVYY4URBXKON7B46JYWCTCFX7AGUA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.493560702 (613h19m29.364643737s) [noise=true , address=145.79.227.46:4603 , piece_id=TH74SWTPYY2RS5HCYD6HVZ7AKT7PECP54QW2O2QBPOGE6KBNZZ3Q, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.493939974 (613h19m29.365023009s) [noise=true , address=45.91.116.7:4010 , piece_id=WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494242320 (613h19m29.365325355s) [noise=true , address=145.79.227.10:4402 , piece_id=ZRVS7X3J5F4SOMWLXMK43YG3B53CY5H2TSBNWD2GUGIUVEBVOIVA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494532452 (613h19m29.365615487s) [noise=true , address=145.79.227.8:4331 , piece_id=VYSOLGYMWJFNNKDMCIO2H4HPP25XSPQR2A2U3DAMW32COJUV56DA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494831212 (613h19m29.365914247s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.495135831 (613h19m29.366218866s) [noise=true , address=145.79.227.4:4186 , piece_id=HLLYRK34RYCEHAEJYB2U4VONMZDQNOUPGV474QWNFJT6JXJXB2QA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.495582452 (613h19m29.366665487s) [noise=true , address=45.91.116.11:4117 , piece_id=QEMAR37LSEBTIWVJ55BE45B72F4EABQD7L4LBZFRUSIPC24X5KMA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.496876726 (613h19m29.367959771s) [noise=true , address=145.79.227.48:4302 , piece_id=HBOPVTDU6TIZHKPTH6TPWMGNNQTBWSR3KVTXCFDHHV2DB2EMEZPQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.497315261 (613h19m29.368398296s) [noise=true , address=77.90.31.16:4735 , piece_id=64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.498610377 (613h19m29.369693412s) [noise=true , address=145.79.227.38:4978 , piece_id=UNIKEWLH6FUFTUNBAV5EYKFMD2BSFEFQM5SCI7T5ATG64OT6UPWA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.500086187 (613h19m29.371169232s) [noise=true , address=145.79.227.20:4437 , piece_id=K72VWZ5ULZTUXF7JYIQP7PHAL66QWYH666DWKRJNEGIHQFEXLGCQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.501835528 (613h19m29.372918563s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.503464039 (613h19m29.374547084s) [noise=true , address=145.79.227.42:4123 , piece_id=UA5ORVLFUIJMB7HEBJLQWITPNJJ5CVRT7DIWX6NR4PFHMVJAEPGQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.506372438 (613h19m29.377455473s) [noise=true , address=145.79.227.18:4007 , piece_id=5NBDLAJ7XXF2AQVMNLUJ5JLTKIPEQFPEDACN5ZTWEMQSHCR7G4UA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:31.208953591 (613h19m31.080036626s) [upload=537472]: Next returned nil segment Dec 31 04:48:31.208969060 (613h19m31.080052105s) [upload=537472]: Waiting for error group managing segments... Dec 31 04:48:31.479989536 (613h19m31.351072581s) [noise=true , address=145.79.227.46:4603 , piece_id=TH74SWTPYY2RS5HCYD6HVZ7AKT7PECP54QW2O2QBPOGE6KBNZZ3Q, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.497388966 (613h19m31.368472012s) [noise=true , address=145.79.227.4:4186 , piece_id=HLLYRK34RYCEHAEJYB2U4VONMZDQNOUPGV474QWNFJT6JXJXB2QA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.499652016 (613h19m31.370735061s) [noise=true , address=145.79.227.8:4331 , piece_id=VYSOLGYMWJFNNKDMCIO2H4HPP25XSPQR2A2U3DAMW32COJUV56DA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.504212910 (613h19m31.375295945s) [noise=true , address=145.79.227.10:4402 , piece_id=ZRVS7X3J5F4SOMWLXMK43YG3B53CY5H2TSBNWD2GUGIUVEBVOIVA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.520135910 (613h19m31.391218955s) [noise=true , address=145.79.227.20:4437 , piece_id=K72VWZ5ULZTUXF7JYIQP7PHAL66QWYH666DWKRJNEGIHQFEXLGCQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.524280291 (613h19m31.395363336s) [noise=true , address=145.79.227.18:4007 , piece_id=5NBDLAJ7XXF2AQVMNLUJ5JLTKIPEQFPEDACN5ZTWEMQSHCR7G4UA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.526111558 (613h19m31.397194593s) [noise=true , address=145.79.227.48:4302 , piece_id=HBOPVTDU6TIZHKPTH6TPWMGNNQTBWSR3KVTXCFDHHV2DB2EMEZPQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.582629986 (613h19m31.453713031s) [noise=true , address=145.79.227.38:4978 , piece_id=UNIKEWLH6FUFTUNBAV5EYKFMD2BSFEFQM5SCI7T5ATG64OT6UPWA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.593131392 (613h19m31.464214437s) [noise=true , address=145.79.227.42:4123 , piece_id=UA5ORVLFUIJMB7HEBJLQWITPNJJ5CVRT7DIWX6NR4PFHMVJAEPGQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.751417148 (613h19m31.622500183s) [noise=true , address=77.90.31.16:4735 , piece_id=64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.819681973 (613h19m31.690765018s) [noise=true , address=45.91.116.7:4010 , piece_id=WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.854844226 (613h19m31.725927261s) [noise=true , address=45.91.116.11:4117 , piece_id=QEMAR37LSEBTIWVJ55BE45B72F4EABQD7L4LBZFRUSIPC24X5KMA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.893142410 (613h19m31.764225495s) [noise=true , address=45.91.116.10:4084 , piece_id=Z7NS7OSJ4ZVKMJA24OPYZJ7LVYY4URBXKON7B46JYWCTCFX7AGUA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.893168720 (613h19m31.764251775s) [seg_pos={0 0}, upload=537472]: Segment reached optimal threshold of 13 pieces. Dec 31 04:48:31.893387546 (613h19m31.764470643s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893397095 (613h19m31.76448015s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893588169 (613h19m31.764671204s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893622604 (613h19m31.764705649s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893764454 (613h19m31.764847499s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893775706 (613h19m31.764858751s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893830881 (613h19m31.764913916s) [seg_pos={0 0}, upload=537472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:48:31.893918268 (613h19m31.765001323s) [upload=537472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:48:31.893924299 (613h19m31.765007334s) [upload=537472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:48:32.194870107 (613h19m32.065953142s) [upload=537472]: Done uploading segments... Dec 31 04:48:32.194894533 (613h19m32.065977569s) [upload=537472]: Upload finished. err: <nil> Duration: 5.821611635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX........X.....X.............X........................] duration: 39.239531121s trace: https://jaeger.dev.storj.io/trace/5943c4171683f71a FAIL (2025-12-31 04:49:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments... Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces. Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments... Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil> Duration: 39.239555759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [........X...X......X...X..X.....X...........X..............X] duration: 5.001472333s trace: https://jaeger.dev.storj.io/trace/7f45acefcc83d80 FAIL (2025-12-31 04:49:40 UTC) orbiter warning: context deadline exceeded Duration: 5.001493162s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 14.953797153s trace: https://jaeger.dev.storj.io/trace/1080f2dcf8b139c9 FAIL (2025-12-31 04:49:38 UTC) orbiter warning: context deadline exceeded Duration: 14.95382151s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X........XX.X...X...X..X...X.......X.......X.........XX.] duration: 5.717308041s trace: https://jaeger.dev.storj.io/trace/7e73fd1b2a67fbfe FAIL (2025-12-31 04:49:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA) context deadline exceeded ======LOGS====== Dec 31 04:48:29.244427553 (613h19m29.115510598s) [upload=537472]: Starting upload Dec 31 04:48:29.244481615 (613h19m29.115564651s) [upload=537472]: Done starting upload Dec 31 04:48:29.244654815 (613h19m29.11573785s) [upload=537472]: Uploading segments... Dec 31 04:48:29.244692277 (613h19m29.115775332s) [upload=537472]: Got next segment. Inline: false Dec 31 04:48:29.244719358 (613h19m29.115802403s) [upload=537472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:48:29.244723957 (613h19m29.115806992s) [upload=537472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:48:29.492669225 (613h19m29.36375226s) [seg_pos={0 0}, upload=537472]: Begin upload segment... Dec 31 04:48:29.492717817 (613h19m29.363800852s) [seg_pos={0 0}, upload=537472]: Done begin upload segment. Dec 31 04:48:29.492750419 (613h19m29.363833454s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.492807709 (613h19m29.363890754s) [noise=true , address=45.91.116.10:4084 , piece_id=Z7NS7OSJ4ZVKMJA24OPYZJ7LVYY4URBXKON7B46JYWCTCFX7AGUA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.493560702 (613h19m29.364643737s) [noise=true , address=145.79.227.46:4603 , piece_id=TH74SWTPYY2RS5HCYD6HVZ7AKT7PECP54QW2O2QBPOGE6KBNZZ3Q, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.493939974 (613h19m29.365023009s) [noise=true , address=45.91.116.7:4010 , piece_id=WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494242320 (613h19m29.365325355s) [noise=true , address=145.79.227.10:4402 , piece_id=ZRVS7X3J5F4SOMWLXMK43YG3B53CY5H2TSBNWD2GUGIUVEBVOIVA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494532452 (613h19m29.365615487s) [noise=true , address=145.79.227.8:4331 , piece_id=VYSOLGYMWJFNNKDMCIO2H4HPP25XSPQR2A2U3DAMW32COJUV56DA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494831212 (613h19m29.365914247s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.495135831 (613h19m29.366218866s) [noise=true , address=145.79.227.4:4186 , piece_id=HLLYRK34RYCEHAEJYB2U4VONMZDQNOUPGV474QWNFJT6JXJXB2QA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.495582452 (613h19m29.366665487s) [noise=true , address=45.91.116.11:4117 , piece_id=QEMAR37LSEBTIWVJ55BE45B72F4EABQD7L4LBZFRUSIPC24X5KMA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.496876726 (613h19m29.367959771s) [noise=true , address=145.79.227.48:4302 , piece_id=HBOPVTDU6TIZHKPTH6TPWMGNNQTBWSR3KVTXCFDHHV2DB2EMEZPQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.497315261 (613h19m29.368398296s) [noise=true , address=77.90.31.16:4735 , piece_id=64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.498610377 (613h19m29.369693412s) [noise=true , address=145.79.227.38:4978 , piece_id=UNIKEWLH6FUFTUNBAV5EYKFMD2BSFEFQM5SCI7T5ATG64OT6UPWA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.500086187 (613h19m29.371169232s) [noise=true , address=145.79.227.20:4437 , piece_id=K72VWZ5ULZTUXF7JYIQP7PHAL66QWYH666DWKRJNEGIHQFEXLGCQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.501835528 (613h19m29.372918563s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.503464039 (613h19m29.374547084s) [noise=true , address=145.79.227.42:4123 , piece_id=UA5ORVLFUIJMB7HEBJLQWITPNJJ5CVRT7DIWX6NR4PFHMVJAEPGQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.506372438 (613h19m29.377455473s) [noise=true , address=145.79.227.18:4007 , piece_id=5NBDLAJ7XXF2AQVMNLUJ5JLTKIPEQFPEDACN5ZTWEMQSHCR7G4UA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:31.208953591 (613h19m31.080036626s) [upload=537472]: Next returned nil segment Dec 31 04:48:31.208969060 (613h19m31.080052105s) [upload=537472]: Waiting for error group managing segments... Dec 31 04:48:31.479989536 (613h19m31.351072581s) [noise=true , address=145.79.227.46:4603 , piece_id=TH74SWTPYY2RS5HCYD6HVZ7AKT7PECP54QW2O2QBPOGE6KBNZZ3Q, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.497388966 (613h19m31.368472012s) [noise=true , address=145.79.227.4:4186 , piece_id=HLLYRK34RYCEHAEJYB2U4VONMZDQNOUPGV474QWNFJT6JXJXB2QA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.499652016 (613h19m31.370735061s) [noise=true , address=145.79.227.8:4331 , piece_id=VYSOLGYMWJFNNKDMCIO2H4HPP25XSPQR2A2U3DAMW32COJUV56DA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.504212910 (613h19m31.375295945s) [noise=true , address=145.79.227.10:4402 , piece_id=ZRVS7X3J5F4SOMWLXMK43YG3B53CY5H2TSBNWD2GUGIUVEBVOIVA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.520135910 (613h19m31.391218955s) [noise=true , address=145.79.227.20:4437 , piece_id=K72VWZ5ULZTUXF7JYIQP7PHAL66QWYH666DWKRJNEGIHQFEXLGCQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.524280291 (613h19m31.395363336s) [noise=true , address=145.79.227.18:4007 , piece_id=5NBDLAJ7XXF2AQVMNLUJ5JLTKIPEQFPEDACN5ZTWEMQSHCR7G4UA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.526111558 (613h19m31.397194593s) [noise=true , address=145.79.227.48:4302 , piece_id=HBOPVTDU6TIZHKPTH6TPWMGNNQTBWSR3KVTXCFDHHV2DB2EMEZPQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.582629986 (613h19m31.453713031s) [noise=true , address=145.79.227.38:4978 , piece_id=UNIKEWLH6FUFTUNBAV5EYKFMD2BSFEFQM5SCI7T5ATG64OT6UPWA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.593131392 (613h19m31.464214437s) [noise=true , address=145.79.227.42:4123 , piece_id=UA5ORVLFUIJMB7HEBJLQWITPNJJ5CVRT7DIWX6NR4PFHMVJAEPGQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.751417148 (613h19m31.622500183s) [noise=true , address=77.90.31.16:4735 , piece_id=64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.819681973 (613h19m31.690765018s) [noise=true , address=45.91.116.7:4010 , piece_id=WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.854844226 (613h19m31.725927261s) [noise=true , address=45.91.116.11:4117 , piece_id=QEMAR37LSEBTIWVJ55BE45B72F4EABQD7L4LBZFRUSIPC24X5KMA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.893142410 (613h19m31.764225495s) [noise=true , address=45.91.116.10:4084 , piece_id=Z7NS7OSJ4ZVKMJA24OPYZJ7LVYY4URBXKON7B46JYWCTCFX7AGUA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.893168720 (613h19m31.764251775s) [seg_pos={0 0}, upload=537472]: Segment reached optimal threshold of 13 pieces. Dec 31 04:48:31.893387546 (613h19m31.764470643s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893397095 (613h19m31.76448015s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893588169 (613h19m31.764671204s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893622604 (613h19m31.764705649s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893764454 (613h19m31.764847499s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893775706 (613h19m31.764858751s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893830881 (613h19m31.764913916s) [seg_pos={0 0}, upload=537472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:48:31.893918268 (613h19m31.765001323s) [upload=537472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:48:31.893924299 (613h19m31.765007334s) [upload=537472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:48:32.194870107 (613h19m32.065953142s) [upload=537472]: Done uploading segments... Dec 31 04:48:32.194894533 (613h19m32.065977569s) [upload=537472]: Upload finished. err: <nil> Duration: 5.717335122s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.184452965s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:49:05 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.184485418s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X........XX.X...X...X..X...X.......X.......X.........XX] duration: 30.002342534s trace: https://jaeger.dev.storj.io/trace/7e73fd1b2a67fbfe FAIL (2025-12-31 04:48:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA) context deadline exceeded ======LOGS====== Dec 31 04:48:29.244427553 (613h19m29.115510598s) [upload=537472]: Starting upload Dec 31 04:48:29.244481615 (613h19m29.115564651s) [upload=537472]: Done starting upload Dec 31 04:48:29.244654815 (613h19m29.11573785s) [upload=537472]: Uploading segments... Dec 31 04:48:29.244692277 (613h19m29.115775332s) [upload=537472]: Got next segment. Inline: false Dec 31 04:48:29.244719358 (613h19m29.115802403s) [upload=537472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:48:29.244723957 (613h19m29.115806992s) [upload=537472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:48:29.492669225 (613h19m29.36375226s) [seg_pos={0 0}, upload=537472]: Begin upload segment... Dec 31 04:48:29.492717817 (613h19m29.363800852s) [seg_pos={0 0}, upload=537472]: Done begin upload segment. Dec 31 04:48:29.492750419 (613h19m29.363833454s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.492807709 (613h19m29.363890754s) [noise=true , address=45.91.116.10:4084 , piece_id=Z7NS7OSJ4ZVKMJA24OPYZJ7LVYY4URBXKON7B46JYWCTCFX7AGUA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.493560702 (613h19m29.364643737s) [noise=true , address=145.79.227.46:4603 , piece_id=TH74SWTPYY2RS5HCYD6HVZ7AKT7PECP54QW2O2QBPOGE6KBNZZ3Q, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.493939974 (613h19m29.365023009s) [noise=true , address=45.91.116.7:4010 , piece_id=WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494242320 (613h19m29.365325355s) [noise=true , address=145.79.227.10:4402 , piece_id=ZRVS7X3J5F4SOMWLXMK43YG3B53CY5H2TSBNWD2GUGIUVEBVOIVA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494532452 (613h19m29.365615487s) [noise=true , address=145.79.227.8:4331 , piece_id=VYSOLGYMWJFNNKDMCIO2H4HPP25XSPQR2A2U3DAMW32COJUV56DA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.494831212 (613h19m29.365914247s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.495135831 (613h19m29.366218866s) [noise=true , address=145.79.227.4:4186 , piece_id=HLLYRK34RYCEHAEJYB2U4VONMZDQNOUPGV474QWNFJT6JXJXB2QA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.495582452 (613h19m29.366665487s) [noise=true , address=45.91.116.11:4117 , piece_id=QEMAR37LSEBTIWVJ55BE45B72F4EABQD7L4LBZFRUSIPC24X5KMA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.496876726 (613h19m29.367959771s) [noise=true , address=145.79.227.48:4302 , piece_id=HBOPVTDU6TIZHKPTH6TPWMGNNQTBWSR3KVTXCFDHHV2DB2EMEZPQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.497315261 (613h19m29.368398296s) [noise=true , address=77.90.31.16:4735 , piece_id=64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.498610377 (613h19m29.369693412s) [noise=true , address=145.79.227.38:4978 , piece_id=UNIKEWLH6FUFTUNBAV5EYKFMD2BSFEFQM5SCI7T5ATG64OT6UPWA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.500086187 (613h19m29.371169232s) [noise=true , address=145.79.227.20:4437 , piece_id=K72VWZ5ULZTUXF7JYIQP7PHAL66QWYH666DWKRJNEGIHQFEXLGCQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.501835528 (613h19m29.372918563s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.503464039 (613h19m29.374547084s) [noise=true , address=145.79.227.42:4123 , piece_id=UA5ORVLFUIJMB7HEBJLQWITPNJJ5CVRT7DIWX6NR4PFHMVJAEPGQ, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:29.506372438 (613h19m29.377455473s) [noise=true , address=145.79.227.18:4007 , piece_id=5NBDLAJ7XXF2AQVMNLUJ5JLTKIPEQFPEDACN5ZTWEMQSHCR7G4UA, seg_pos={0 0}, upload=537472]: Uploading piece... Dec 31 04:48:31.208953591 (613h19m31.080036626s) [upload=537472]: Next returned nil segment Dec 31 04:48:31.208969060 (613h19m31.080052105s) [upload=537472]: Waiting for error group managing segments... Dec 31 04:48:31.479989536 (613h19m31.351072581s) [noise=true , address=145.79.227.46:4603 , piece_id=TH74SWTPYY2RS5HCYD6HVZ7AKT7PECP54QW2O2QBPOGE6KBNZZ3Q, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.497388966 (613h19m31.368472012s) [noise=true , address=145.79.227.4:4186 , piece_id=HLLYRK34RYCEHAEJYB2U4VONMZDQNOUPGV474QWNFJT6JXJXB2QA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.499652016 (613h19m31.370735061s) [noise=true , address=145.79.227.8:4331 , piece_id=VYSOLGYMWJFNNKDMCIO2H4HPP25XSPQR2A2U3DAMW32COJUV56DA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.504212910 (613h19m31.375295945s) [noise=true , address=145.79.227.10:4402 , piece_id=ZRVS7X3J5F4SOMWLXMK43YG3B53CY5H2TSBNWD2GUGIUVEBVOIVA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.520135910 (613h19m31.391218955s) [noise=true , address=145.79.227.20:4437 , piece_id=K72VWZ5ULZTUXF7JYIQP7PHAL66QWYH666DWKRJNEGIHQFEXLGCQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.524280291 (613h19m31.395363336s) [noise=true , address=145.79.227.18:4007 , piece_id=5NBDLAJ7XXF2AQVMNLUJ5JLTKIPEQFPEDACN5ZTWEMQSHCR7G4UA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.526111558 (613h19m31.397194593s) [noise=true , address=145.79.227.48:4302 , piece_id=HBOPVTDU6TIZHKPTH6TPWMGNNQTBWSR3KVTXCFDHHV2DB2EMEZPQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.582629986 (613h19m31.453713031s) [noise=true , address=145.79.227.38:4978 , piece_id=UNIKEWLH6FUFTUNBAV5EYKFMD2BSFEFQM5SCI7T5ATG64OT6UPWA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.593131392 (613h19m31.464214437s) [noise=true , address=145.79.227.42:4123 , piece_id=UA5ORVLFUIJMB7HEBJLQWITPNJJ5CVRT7DIWX6NR4PFHMVJAEPGQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.751417148 (613h19m31.622500183s) [noise=true , address=77.90.31.16:4735 , piece_id=64YFBL5JRPYLWO3ZKMRILTQZACCDNDMWJU5S6NFJXJQFHSDNHPIA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.819681973 (613h19m31.690765018s) [noise=true , address=45.91.116.7:4010 , piece_id=WQLYOJHHR556DPDG7WR4EBVO2BIYOB75XKGXQ3R53R4WNGHAPCTQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.854844226 (613h19m31.725927261s) [noise=true , address=45.91.116.11:4117 , piece_id=QEMAR37LSEBTIWVJ55BE45B72F4EABQD7L4LBZFRUSIPC24X5KMA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.893142410 (613h19m31.764225495s) [noise=true , address=45.91.116.10:4084 , piece_id=Z7NS7OSJ4ZVKMJA24OPYZJ7LVYY4URBXKON7B46JYWCTCFX7AGUA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: <nil> Dec 31 04:48:31.893168720 (613h19m31.764251775s) [seg_pos={0 0}, upload=537472]: Segment reached optimal threshold of 13 pieces. Dec 31 04:48:31.893387546 (613h19m31.764470643s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893397095 (613h19m31.76448015s) [noise=true , address=45.91.116.5:4299 , piece_id=P7NEFNIPW2IOSKPGKGHETXGYGP4XKJDWSFEDN6DFJAU3XMVS7GKA, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893588169 (613h19m31.764671204s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893622604 (613h19m31.764705649s) [noise=true , address=45.91.116.9:4261 , piece_id=5RJ6SOPUYQPKCKCVQJPTCBKBGICBFUKVD4J25HIMHL65SDJRR2EA, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893764454 (613h19m31.764847499s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 31 04:48:31.893775706 (613h19m31.764858751s) [noise=true , address=45.91.116.12:4227 , piece_id=H56OHSPDNP34QTJ264YCZLT5ROIWVUM6F7Z27OGJFPBLHT634ZYQ, seg_pos={0 0}, upload=537472]: Long tail cancellation due to optimal threshold Dec 31 04:48:31.893830881 (613h19m31.764913916s) [seg_pos={0 0}, upload=537472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:48:31.893918268 (613h19m31.765001323s) [upload=537472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:48:31.893924299 (613h19m31.765007334s) [upload=537472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:48:32.194870107 (613h19m32.065953142s) [upload=537472]: Done uploading segments... Dec 31 04:48:32.194894533 (613h19m32.065977569s) [upload=537472]: Upload finished. err: <nil> Duration: 30.002368673s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 15.922539005s trace: https://jaeger.dev.storj.io/trace/1080f2dcf8b139c9 FAIL (2025-12-31 04:48:56 UTC) orbiter warning: context deadline exceeded Duration: 15.922564364s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XX........X.....X.............X.......................] duration: 41.244787796s trace: https://jaeger.dev.storj.io/trace/5943c4171683f71a FAIL (2025-12-31 04:48:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments... Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces. Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments... Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil> Duration: 41.244816551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.322813981s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:48:43 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.322849447s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........X...X......X...X..X.....X...........X..............] duration: 3.626403567s trace: https://jaeger.dev.storj.io/trace/34423987b8e10d3b FAIL (2025-12-31 04:48:32 UTC) orbiter warning: context deadline exceeded Duration: 3.626429166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 19.719299818s trace: https://jaeger.dev.storj.io/trace/1080f2dcf8b139c9 FAIL (2025-12-31 04:48:16 UTC) orbiter warning: context deadline exceeded Duration: 19.719328933s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X.........X...X......X...X..X.....X...........X.............] duration: 3.571218721s trace: https://jaeger.dev.storj.io/trace/34423987b8e10d3b FAIL (2025-12-31 04:48:11 UTC) orbiter warning: context deadline exceeded Duration: 3.571248627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XX........X.....X.............X......................] duration: 27.770134983s trace: https://jaeger.dev.storj.io/trace/5943c4171683f71a FAIL (2025-12-31 04:48:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments... Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces. Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments... Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil> Duration: 27.770160171s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X........XX.X...X...X..X...X.......X.......X.........X] duration: 30.0051991s trace: https://jaeger.dev.storj.io/trace/1a1c0eac735801 FAIL (2025-12-31 04:48:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: IPB6N4YFEUL5QRJINYLJM3PF3SR2P4R6MWPDJ6BQWAAO4H7FWM3A) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: B7WG5NCZB3IFYPXVBJJ2VPC2GGYB6I5A4QZAH2XZ4TCUFSFGM4NQ) context deadline exceeded ======LOGS====== Dec 31 04:47:32.039978083 (613h18m31.911061118s) [upload=537459]: Starting upload Dec 31 04:47:32.040053106 (613h18m31.911136161s) [upload=537459]: Done starting upload Dec 31 04:47:32.040725157 (613h18m31.911808202s) [upload=537459]: Uploading segments... Dec 31 04:47:32.040786943 (613h18m31.911869978s) [upload=537459]: Got next segment. Inline: false Dec 31 04:47:32.040846687 (613h18m31.911929733s) [upload=537459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:47:32.040852719 (613h18m31.911935764s) [upload=537459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:47:32.285950477 (613h18m32.157033512s) [seg_pos={0 0}, upload=537459]: Begin upload segment... Dec 31 04:47:32.286000924 (613h18m32.157083969s) [seg_pos={0 0}, upload=537459]: Done begin upload segment. Dec 31 04:47:32.286040599 (613h18m32.157123644s) [noise=true , address=45.91.116.7:4012 , piece_id=F66NIH4PCWZZTJ6G6M6AEVPCMYZSVU6NAYKD5UXK5D6INMA4MPRQ, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.286186778 (613h18m32.157269823s) [noise=true , address=145.79.227.32:4836 , piece_id=OZ7MD7JF3S7LH2XROCIQBXEBHSMYYM7I4236LAUHVXYGPU6YBHYQ, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.286857014 (613h18m32.157940049s) [noise=true , address=145.79.227.39:4013 , piece_id=JMLYXWBIDP3NDJMHYWRTFSUY7KPZM2ZHMXNNOIQM2SFW46TJ2AEA, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.287076522 (613h18m32.158159557s) [noise=true , address=77.90.31.19:4842 , piece_id=IPB6N4YFEUL5QRJINYLJM3PF3SR2P4R6MWPDJ6BQWAAO4H7FWM3A, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.287579810 (613h18m32.158662845s) [noise=true , address=145.79.227.24:4583 , piece_id=S4EIUI2P2COJGGF77APCYJHDUXVJZPN6ZUP2K2YADKYXGCNURUKA, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.287931880 (613h18m32.159014915s) [noise=true , address=145.79.227.6:4258 , piece_id=ZQ7F2PIFSJQI3Z7FRNS7GMXLNYXVWLLYWXYTNR3DFBSLMBETABJQ, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.288265406 (613h18m32.159348441s) [noise=true , address=77.90.31.12:4590 , piece_id=B7WG5NCZB3IFYPXVBJJ2VPC2GGYB6I5A4QZAH2XZ4TCUFSFGM4NQ, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.288658574 (613h18m32.159741609s) [noise=true , address=145.79.227.13:4802 , piece_id=INZT747LU2E4LHJCP5S4GTR6VGKKVZR3UCWJHEIL4PPFIUQZT2IA, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.289147124 (613h18m32.16023016s) [noise=true , address=145.79.227.7:4294 , piece_id=DTXEWC3CPJIJENIMWJFD7CKGLLSQZ2TFQOQE4SGNFZKA3J6IICHA, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.289408994 (613h18m32.160492029s) [noise=true , address=145.79.227.51:4411 , piece_id=CLDIUN454OMGPVY74UYINX65QVQGYBWJ6CZVZKWLQSXPEAN4HCGA, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.289789398 (613h18m32.160872433s) [noise=true , address=45.91.116.4:4939 , piece_id=LWCRSLTD55LLQHHBCFQJLWBS3YEWV5U47IU6XWCHBZIOKXS7DAIQ, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.290039644 (613h18m32.161122699s) [noise=true , address=145.79.227.16:4911 , piece_id=DHETE7YKWXN6S7H5RSGVRFLHLY5ENGL53COTFURUJ2APSBGDHM5A, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.290442441 (613h18m32.161525476s) [noise=true , address=145.79.227.28:4726 , piece_id=AUZX2SLAA6AQHHQF33JGMYKD6IWV6KFFN2DQGKNY2DKRWTHQCJ7Q, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.290835520 (613h18m32.161918575s) [noise=true , address=45.91.116.10:4083 , piece_id=US57AFNSAU3Y2UYENFJUD5374B3VM2BGKFD7LVISB3UXE2SJBAUQ, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.291265889 (613h18m32.162348924s) [noise=true , address=45.91.116.6:4976 , piece_id=22EKLN3S62RUI6NOU2OSKPLG4VQC47VURIPYDM7HZ6DGLIZK25TQ, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:32.292247558 (613h18m32.163330593s) [noise=true , address=145.79.227.41:4087 , piece_id=EOH3FFUVK7U75PWONHQKZGNIIPAWUK7MHZOXBSHXD2QHVKKLAJTA, seg_pos={0 0}, upload=537459]: Uploading piece... Dec 31 04:47:33.992592573 (613h18m33.863675618s) [upload=537459]: Next returned nil segment Dec 31 04:47:33.992608854 (613h18m33.863691899s) [upload=537459]: Waiting for error group managing segments... Dec 31 04:47:34.275691539 (613h18m34.146774584s) [noise=true , address=145.79.227.6:4258 , piece_id=ZQ7F2PIFSJQI3Z7FRNS7GMXLNYXVWLLYWXYTNR3DFBSLMBETABJQ, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.275789575 (613h18m34.14687262s) [noise=true , address=145.79.227.28:4726 , piece_id=AUZX2SLAA6AQHHQF33JGMYKD6IWV6KFFN2DQGKNY2DKRWTHQCJ7Q, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.277723418 (613h18m34.148806463s) [noise=true , address=145.79.227.16:4911 , piece_id=DHETE7YKWXN6S7H5RSGVRFLHLY5ENGL53COTFURUJ2APSBGDHM5A, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.291382436 (613h18m34.162465501s) [noise=true , address=145.79.227.39:4013 , piece_id=JMLYXWBIDP3NDJMHYWRTFSUY7KPZM2ZHMXNNOIQM2SFW46TJ2AEA, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.298065221 (613h18m34.169148256s) [noise=true , address=145.79.227.13:4802 , piece_id=INZT747LU2E4LHJCP5S4GTR6VGKKVZR3UCWJHEIL4PPFIUQZT2IA, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.300247917 (613h18m34.171330962s) [noise=true , address=145.79.227.41:4087 , piece_id=EOH3FFUVK7U75PWONHQKZGNIIPAWUK7MHZOXBSHXD2QHVKKLAJTA, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.300404415 (613h18m34.17148745s) [noise=true , address=145.79.227.51:4411 , piece_id=CLDIUN454OMGPVY74UYINX65QVQGYBWJ6CZVZKWLQSXPEAN4HCGA, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.301536982 (613h18m34.172620017s) [noise=true , address=145.79.227.32:4836 , piece_id=OZ7MD7JF3S7LH2XROCIQBXEBHSMYYM7I4236LAUHVXYGPU6YBHYQ, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.359423545 (613h18m34.23050658s) [noise=true , address=145.79.227.7:4294 , piece_id=DTXEWC3CPJIJENIMWJFD7CKGLLSQZ2TFQOQE4SGNFZKA3J6IICHA, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.362589143 (613h18m34.233672188s) [noise=true , address=145.79.227.24:4583 , piece_id=S4EIUI2P2COJGGF77APCYJHDUXVJZPN6ZUP2K2YADKYXGCNURUKA, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.453350724 (613h18m34.324433759s) [noise=true , address=77.90.31.19:4842 , piece_id=IPB6N4YFEUL5QRJINYLJM3PF3SR2P4R6MWPDJ6BQWAAO4H7FWM3A, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.547995760 (613h18m34.419078795s) [noise=true , address=77.90.31.12:4590 , piece_id=B7WG5NCZB3IFYPXVBJJ2VPC2GGYB6I5A4QZAH2XZ4TCUFSFGM4NQ, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.593354068 (613h18m34.464437113s) [noise=true , address=45.91.116.10:4083 , piece_id=US57AFNSAU3Y2UYENFJUD5374B3VM2BGKFD7LVISB3UXE2SJBAUQ, seg_pos={0 0}, upload=537459]: Done uploading piece. err: <nil> Dec 31 04:47:34.593378093 (613h18m34.464461128s) [seg_pos={0 0}, upload=537459]: Segment reached optimal threshold of 13 pieces. Dec 31 04:47:34.593681581 (613h18m34.464764626s) [noise=true , address=45.91.116.6:4976 , piece_id=22EKLN3S62RUI6NOU2OSKPLG4VQC47VURIPYDM7HZ6DGLIZK25TQ, seg_pos={0 0}, upload=537459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 31 04:47:34.593692602 (613h18m34.464775637s) [noise=true , address=45.91.116.6:4976 , piece_id=22EKLN3S62RUI6NOU2OSKPLG4VQC47VURIPYDM7HZ6DGLIZK25TQ, seg_pos={0 0}, upload=537459]: Long tail cancellation due to optimal threshold Dec 31 04:47:34.593708443 (613h18m34.464791478s) [noise=true , address=45.91.116.7:4012 , piece_id=F66NIH4PCWZZTJ6G6M6AEVPCMYZSVU6NAYKD5UXK5D6INMA4MPRQ, seg_pos={0 0}, upload=537459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 31 04:47:34.593723832 (613h18m34.464806877s) [noise=true , address=45.91.116.7:4012 , piece_id=F66NIH4PCWZZTJ6G6M6AEVPCMYZSVU6NAYKD5UXK5D6INMA4MPRQ, seg_pos={0 0}, upload=537459]: Long tail cancellation due to optimal threshold Dec 31 04:47:34.594044232 (613h18m34.465127278s) [noise=true , address=45.91.116.4:4939 , piece_id=LWCRSLTD55LLQHHBCFQJLWBS3YEWV5U47IU6XWCHBZIOKXS7DAIQ, seg_pos={0 0}, upload=537459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:37954->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 04:47:34.594053129 (613h18m34.465136174s) [noise=true , address=45.91.116.4:4939 , piece_id=LWCRSLTD55LLQHHBCFQJLWBS3YEWV5U47IU6XWCHBZIOKXS7DAIQ, seg_pos={0 0}, upload=537459]: Long tail cancellation due to optimal threshold Dec 31 04:47:34.594073588 (613h18m34.465156623s) [seg_pos={0 0}, upload=537459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:47:34.594150365 (613h18m34.46523341s) [upload=537459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:47:34.594155454 (613h18m34.465238499s) [upload=537459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:47:34.936048661 (613h18m34.807131696s) [upload=537459]: Done uploading segments... Dec 31 04:47:34.936066595 (613h18m34.80714964s) [upload=537459]: Upload finished. err: <nil> Duration: 30.005224399s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 4.924589785s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:47:52 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 4.92461852s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X.........X...X......X...X..X.....X...........X............] duration: 3.94950342s trace: https://jaeger.dev.storj.io/trace/34423987b8e10d3b FAIL (2025-12-31 04:47:38 UTC) orbiter warning: context deadline exceeded Duration: 3.949528038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.227034599s trace: https://jaeger.dev.storj.io/trace/3d2998323a09f583 FAIL (2025-12-31 04:47:28 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.227056911s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 7.283991174s trace: https://jaeger.dev.storj.io/trace/1080f2dcf8b139c9 FAIL (2025-12-31 04:47:25 UTC) orbiter warning: context deadline exceeded Duration: 7.284020571s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XX........X.....X.............X.....................] duration: 45.635523576s trace: https://jaeger.dev.storj.io/trace/5943c4171683f71a FAIL (2025-12-31 04:47:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.561773894 (612h59m14.432856939s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562188874 (612h59m14.433271919s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562325113 (612h59m14.433408159s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562700307 (612h59m14.433783352s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.562915267 (612h59m14.433998312s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563378759 (612h59m14.434461795s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.563676125 (612h59m14.43475916s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564463114 (612h59m14.435546149s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.564699855 (612h59m14.4357829s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565422932 (612h59m14.436505987s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.565963431 (612h59m14.437046476s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.566356520 (612h59m14.437439555s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568289922 (612h59m14.439372967s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.568768132 (612h59m14.439851167s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:14.569380038 (612h59m14.440463083s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Uploading piece... Dec 31 04:28:15.986614614 (612h59m15.857697659s) [upload=537172]: Next returned nil segment Dec 31 04:28:15.986640534 (612h59m15.857723589s) [upload=537172]: Waiting for error group managing segments... Dec 31 04:28:16.490756260 (612h59m16.361839295s) [noise=true , address=145.79.227.43:4174 , piece_id=BIWSD3XUL4JNIGU5U6XKD3EOAD2G4NHPVZAS4JH3I6RZTRV6IMPA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.611981168 (612h59m16.483064203s) [noise=true , address=77.90.31.15:4712 , piece_id=XHP67ASX4OSV7VUJ325VJYCSJK55VE5RCPNWLIG674EZUJWEITAA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.614186197 (612h59m16.485269242s) [noise=true , address=145.79.227.33:4884 , piece_id=IUMNLH2UF4QOPBYRTYMNYHI32SLIK7K34TRNMYKNBDJPOLFMRQFA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.685725660 (612h59m16.556808695s) [noise=true , address=77.90.31.18:4814 , piece_id=RYXAWZVLWZ3UGUDMNF7HYTEUQ2ML3DX4GRHACU6SQTWVS6267SOA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.829877057 (612h59m16.700960092s) [noise=true , address=45.91.116.10:4087 , piece_id=A2IV6Q7BCWC2DBY2INMOGCP6C2D2VEFPJUGZ24SD7AOEG2Y2ELUA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.852577064 (612h59m16.72366011s) [noise=true , address=77.90.31.12:4611 , piece_id=4F73UG7G5SU5HSIN2MHWAA65BARICSYFXNYBWE2AADFVZ5TLMR2Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.881952174 (612h59m16.753035209s) [noise=true , address=45.91.116.13:4162 , piece_id=EYKBO2BHGEWIVKLS5MYVXRPOMQB7NQWWRTIPQCWPTTJC2R4IP2KQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.895465688 (612h59m16.766548733s) [noise=true , address=45.91.116.14:4218 , piece_id=R6FDT75336ECWKQ5G7UFOIOAM4AIPSTFXMXFYY5SPHLPNKZ7NB3A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:16.944421594 (612h59m16.815504639s) [noise=true , address=145.79.227.40:4073 , piece_id=OT5EIBQATWX6SQVUWIDSEP4TGLCDQ6M2C4C3UXGLPFAMPFAPCFQA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.011647332 (612h59m16.882730377s) [noise=true , address=77.90.31.21:4898 , piece_id=IBY7MZTLPZUC6YFQVAXQBFNTJHWTDOC7XFJUN2FAGSFM56CGBTPQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.247944632 (612h59m17.119027677s) [noise=true , address=193.149.164.143:4701 , piece_id=CXFAP52AYSIFKO35TXD2INGUNBWYLMUUHP77GN6L47KGHR7UR65Q, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.312279154 (612h59m17.183362189s) [noise=true , address=145.79.227.30:4788 , piece_id=BO4PEHE2UNJQUAG46UOVFAPSVBBUXSZK6Z567MXU562CRH5FGL2A, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.420002501 (612h59m17.291085546s) [noise=true , address=145.79.227.34:4933 , piece_id=HH6W4LSJ4EVGKSXQLXQVB3YMIWIXHPRILG72IO5MWQ3TDKMQIXHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496274376 (612h59m17.367357421s) [noise=true , address=145.79.227.16:4921 , piece_id=BDFWPIHVFVXD57BQWJDOW4LSWFUUTLF4BSV4B2DTXV45PZLA33AA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: <nil> Dec 31 04:28:17.496354528 (612h59m17.367437573s) [seg_pos={0 0}, upload=537172]: Segment reached optimal threshold of 14 pieces. Dec 31 04:28:17.496690268 (612h59m17.367773313s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56804->64.49.13.169:4705: use of closed network connection Dec 31 04:28:17.496702401 (612h59m17.367785446s) [noise=true , address=64.49.13.169:4705 , piece_id=ETZV24XSZC33G4GNUHA3WJQM2MNDVQTJS7PCB6LX5L64IJYA25PQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496719914 (612h59m17.367802949s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496737698 (612h59m17.367820733s) [noise=true , address=77.90.31.13:4654 , piece_id=B26GOHLGDBW7SRQSEX76IEY2XXBUMHUEYQ2CREEVVS4R4AX25YSA, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.496939051 (612h59m17.368022086s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.496946707 (612h59m17.368029742s) [noise=true , address=193.149.164.143:4697 , piece_id=AFLDOWJXZLWLC4NTPCXWWN6JYE5JQJNZ5GVE6KNBW434GX636FHQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497321339 (612h59m17.368404374s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 31 04:28:17.497331189 (612h59m17.368414244s) [noise=true , address=45.91.116.11:4144 , piece_id=MYMCLTSBXMNUU4VIAR3CTEBNYONMBW5E3TFCA72RTGVTTPBDQ4UQ, seg_pos={0 0}, upload=537172]: Long tail cancellation due to optimal threshold Dec 31 04:28:17.497361846 (612h59m17.368444891s) [seg_pos={0 0}, upload=537172]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 04:28:17.497453712 (612h59m17.368536747s) [upload=537172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:28:17.497459563 (612h59m17.368542608s) [upload=537172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:28:17.787387142 (612h59m17.658470177s) [upload=537172]: Done uploading segments... Dec 31 04:28:17.787406819 (612h59m17.658489854s) [upload=537172]: Upload finished. err: <nil> Duration: 45.635558832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X......X........XX.X...X...X..X...X.......X.......X.........] duration: 5.763314272s trace: https://jaeger.dev.storj.io/trace/3f8f3b407299da9 FAIL (2025-12-31 04:46:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 44B3JBKREOSLD57UQLATVRSWGJM7JG7I2OUQ3ZBZIEWO4C5SX6QQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JAWGUNYPYTC2KXAJJTOKVTABC6E3QBZRJOCI3WOW3YQH5VYXTBVQ) context deadline exceeded ======LOGS====== Dec 31 04:41:55.331791392 (613h12m55.202874427s) [upload=537372]: Starting upload Dec 31 04:41:55.331846657 (613h12m55.202929692s) [upload=537372]: Done starting upload Dec 31 04:41:55.332042339 (613h12m55.203125374s) [upload=537372]: Uploading segments... Dec 31 04:41:55.332135437 (613h12m55.203218482s) [upload=537372]: Got next segment. Inline: false Dec 31 04:41:55.332174191 (613h12m55.203257226s) [upload=537372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:41:55.332179611 (613h12m55.203262646s) [upload=537372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:41:55.580379825 (613h12m55.45146286s) [seg_pos={0 0}, upload=537372]: Begin upload segment... Dec 31 04:41:55.580551632 (613h12m55.451634667s) [seg_pos={0 0}, upload=537372]: Done begin upload segment. Dec 31 04:41:55.580626795 (613h12m55.45170984s) [noise=true , address=45.91.116.6:4975 , piece_id=JAWGUNYPYTC2KXAJJTOKVTABC6E3QBZRJOCI3WOW3YQH5VYXTBVQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.581826018 (613h12m55.452909053s) [noise=true , address=45.91.116.7:4009 , piece_id=P2KQXKZJDY43VTVO7KW4PFQBP3NEL42SLSLPKFAYAUF4XXZ73OLQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.581860834 (613h12m55.4529439s) [noise=true , address=145.79.227.47:4265 , piece_id=JI4H7VUUERKAAX2Y6USZU2FTGSLU7GVPSNT73TXKIZZANBL5EDWA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.582139415 (613h12m55.45322246s) [noise=true , address=45.91.116.10:4081 , piece_id=XEGUJAS64MBGEYPSRMICVZ36R7OW4CGKHCYOXRFXIPGVFHHTG2FA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.582410030 (613h12m55.453493066s) [noise=true , address=145.79.227.41:4086 , piece_id=NDEO3MG5XY37OYTH6F6FNZGPENAAPIPF66FGETX6SSFTMEVWMMCQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.582531452 (613h12m55.453614487s) [noise=true , address=145.79.227.50:4373 , piece_id=R4HJ3LANTIYXX2PFW5YSPNAZU5RSGYOH536VN54IN4W2FJWKJZIA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.582736252 (613h12m55.453819287s) [noise=true , address=77.90.31.11:4556 , piece_id=3YEXD7HCKDVX4YIRGNZBYIU4ERD5W36DFCGZHESP3PRIGDPZP7TQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.582951672 (613h12m55.454034717s) [noise=true , address=145.79.227.25:4617 , piece_id=DIEL3BTDZIWUYVGYXLOSLJEEGGKRB7MKSVDYC3NHGCKWHIDZUDZQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.583092561 (613h12m55.454175626s) [noise=true , address=145.79.227.28:4726 , piece_id=ZYYILSQVLETAUKBJJY2S6AQ2IWG4CERFT3MWPEOUNV2SHYQ5AFAA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.583467485 (613h12m55.45455052s) [noise=true , address=145.79.227.19:4078 , piece_id=NLTLXPSPLV4TWE6KLGX5XSCK6HLQPM55Z2BWOIHMQZXXUEDII7RA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.583474358 (613h12m55.454557403s) [noise=true , address=77.90.31.13:4629 , piece_id=44B3JBKREOSLD57UQLATVRSWGJM7JG7I2OUQ3ZBZIEWO4C5SX6QQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.583773216 (613h12m55.454856251s) [noise=true , address=145.79.227.37:4566 , piece_id=JARGBZWEL4UD76EEISZEFMJ5SWY7RUAULFSWR7STTKM4RMW4FYHQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.584038291 (613h12m55.455121336s) [noise=true , address=145.79.227.48:4303 , piece_id=26ZXX6PUMFGVDOOGWP2IUMDTHIVPBHG4EVHYOH5IDVGSLPNTYEMA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.584120858 (613h12m55.455203903s) [noise=true , address=145.79.227.27:4691 , piece_id=7N4LYT4FTRXI6JV3NI3L65U6NUAUMXR2ZH5FZIMUXKBNRTNNIUEA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.584501463 (613h12m55.455584498s) [noise=true , address=45.91.116.5:4300 , piece_id=NOG7354BIZCG5JG6STFE2PAX7KNIOCZEFLQLDVISEHX24KKSBUAA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.584856520 (613h12m55.455939555s) [noise=true , address=145.79.227.46:4603 , piece_id=BZUHK45GEASBFGGHSFAWFXN5VDF2VFPJ7DNPSZDH6FFKYSSUXR6A, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:57.283478314 (613h12m57.154561349s) [upload=537372]: Next returned nil segment Dec 31 04:41:57.283498974 (613h12m57.154582009s) [upload=537372]: Waiting for error group managing segments... Dec 31 04:41:57.568671938 (613h12m57.439754973s) [noise=true , address=145.79.227.50:4373 , piece_id=R4HJ3LANTIYXX2PFW5YSPNAZU5RSGYOH536VN54IN4W2FJWKJZIA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.568784803 (613h12m57.439867838s) [noise=true , address=145.79.227.46:4603 , piece_id=BZUHK45GEASBFGGHSFAWFXN5VDF2VFPJ7DNPSZDH6FFKYSSUXR6A, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.569149196 (613h12m57.440232231s) [noise=true , address=145.79.227.19:4078 , piece_id=NLTLXPSPLV4TWE6KLGX5XSCK6HLQPM55Z2BWOIHMQZXXUEDII7RA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.577960875 (613h12m57.44904391s) [noise=true , address=145.79.227.37:4566 , piece_id=JARGBZWEL4UD76EEISZEFMJ5SWY7RUAULFSWR7STTKM4RMW4FYHQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.579194625 (613h12m57.45027766s) [noise=true , address=145.79.227.41:4086 , piece_id=NDEO3MG5XY37OYTH6F6FNZGPENAAPIPF66FGETX6SSFTMEVWMMCQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.585963744 (613h12m57.457046789s) [noise=true , address=145.79.227.28:4726 , piece_id=ZYYILSQVLETAUKBJJY2S6AQ2IWG4CERFT3MWPEOUNV2SHYQ5AFAA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.587889711 (613h12m57.458972756s) [noise=true , address=145.79.227.25:4617 , piece_id=DIEL3BTDZIWUYVGYXLOSLJEEGGKRB7MKSVDYC3NHGCKWHIDZUDZQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.600165486 (613h12m57.471248521s) [noise=true , address=145.79.227.48:4303 , piece_id=26ZXX6PUMFGVDOOGWP2IUMDTHIVPBHG4EVHYOH5IDVGSLPNTYEMA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.601065660 (613h12m57.472148705s) [noise=true , address=145.79.227.47:4265 , piece_id=JI4H7VUUERKAAX2Y6USZU2FTGSLU7GVPSNT73TXKIZZANBL5EDWA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.628124203 (613h12m57.499207248s) [noise=true , address=145.79.227.27:4691 , piece_id=7N4LYT4FTRXI6JV3NI3L65U6NUAUMXR2ZH5FZIMUXKBNRTNNIUEA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.818341791 (613h12m57.689424826s) [noise=true , address=77.90.31.13:4629 , piece_id=44B3JBKREOSLD57UQLATVRSWGJM7JG7I2OUQ3ZBZIEWO4C5SX6QQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.868995641 (613h12m57.740078676s) [noise=true , address=77.90.31.11:4556 , piece_id=3YEXD7HCKDVX4YIRGNZBYIU4ERD5W36DFCGZHESP3PRIGDPZP7TQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:58.089298215 (613h12m57.96038125s) [noise=true , address=45.91.116.6:4975 , piece_id=JAWGUNYPYTC2KXAJJTOKVTABC6E3QBZRJOCI3WOW3YQH5VYXTBVQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:58.089317181 (613h12m57.960400216s) [seg_pos={0 0}, upload=537372]: Segment reached optimal threshold of 13 pieces. Dec 31 04:41:58.089524216 (613h12m57.960607251s) [noise=true , address=45.91.116.10:4081 , piece_id=XEGUJAS64MBGEYPSRMICVZ36R7OW4CGKHCYOXRFXIPGVFHHTG2FA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 04:41:58.089531209 (613h12m57.960614254s) [noise=true , address=45.91.116.10:4081 , piece_id=XEGUJAS64MBGEYPSRMICVZ36R7OW4CGKHCYOXRFXIPGVFHHTG2FA, seg_pos={0 0}, upload=537372]: Long tail cancellation due to optimal threshold Dec 31 04:41:58.089694680 (613h12m57.960777715s) [noise=true , address=45.91.116.5:4300 , piece_id=NOG7354BIZCG5JG6STFE2PAX7KNIOCZEFLQLDVISEHX24KKSBUAA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 31 04:41:58.089714317 (613h12m57.960797352s) [noise=true , address=45.91.116.5:4300 , piece_id=NOG7354BIZCG5JG6STFE2PAX7KNIOCZEFLQLDVISEHX24KKSBUAA, seg_pos={0 0}, upload=537372]: Long tail cancellation due to optimal threshold Dec 31 04:41:58.089755436 (613h12m57.960838471s) [noise=true , address=45.91.116.7:4009 , piece_id=P2KQXKZJDY43VTVO7KW4PFQBP3NEL42SLSLPKFAYAUF4XXZ73OLQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 31 04:41:58.089763531 (613h12m57.960846567s) [noise=true , address=45.91.116.7:4009 , piece_id=P2KQXKZJDY43VTVO7KW4PFQBP3NEL42SLSLPKFAYAUF4XXZ73OLQ, seg_pos={0 0}, upload=537372]: Long tail cancellation due to optimal threshold Dec 31 04:41:58.089867910 (613h12m57.960950955s) [seg_pos={0 0}, upload=537372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:41:58.089935529 (613h12m57.961018564s) [upload=537372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:41:58.089940118 (613h12m57.961023153s) [upload=537372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:41:58.388150457 (613h12m58.259233502s) [upload=537372]: Done uploading segments... Dec 31 04:41:58.388168983 (613h12m58.259252018s) [upload=537372]: Upload finished. err: <nil> Duration: 5.763349481s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 10.361806947s trace: https://jaeger.dev.storj.io/trace/1080f2dcf8b139c9 FAIL (2025-12-31 04:46:45 UTC) orbiter warning: context deadline exceeded Duration: 10.361841363s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X......X........XX.X...X...X..X...X.......X.......X........] duration: 6.10229887s trace: https://jaeger.dev.storj.io/trace/3f8f3b407299da9 FAIL (2025-12-31 04:46:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 44B3JBKREOSLD57UQLATVRSWGJM7JG7I2OUQ3ZBZIEWO4C5SX6QQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JAWGUNYPYTC2KXAJJTOKVTABC6E3QBZRJOCI3WOW3YQH5VYXTBVQ) context deadline exceeded ======LOGS====== Dec 31 04:41:55.331791392 (613h12m55.202874427s) [upload=537372]: Starting upload Dec 31 04:41:55.331846657 (613h12m55.202929692s) [upload=537372]: Done starting upload Dec 31 04:41:55.332042339 (613h12m55.203125374s) [upload=537372]: Uploading segments... Dec 31 04:41:55.332135437 (613h12m55.203218482s) [upload=537372]: Got next segment. Inline: false Dec 31 04:41:55.332174191 (613h12m55.203257226s) [upload=537372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 04:41:55.332179611 (613h12m55.203262646s) [upload=537372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 04:41:55.580379825 (613h12m55.45146286s) [seg_pos={0 0}, upload=537372]: Begin upload segment... Dec 31 04:41:55.580551632 (613h12m55.451634667s) [seg_pos={0 0}, upload=537372]: Done begin upload segment. Dec 31 04:41:55.580626795 (613h12m55.45170984s) [noise=true , address=45.91.116.6:4975 , piece_id=JAWGUNYPYTC2KXAJJTOKVTABC6E3QBZRJOCI3WOW3YQH5VYXTBVQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.581826018 (613h12m55.452909053s) [noise=true , address=45.91.116.7:4009 , piece_id=P2KQXKZJDY43VTVO7KW4PFQBP3NEL42SLSLPKFAYAUF4XXZ73OLQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.581860834 (613h12m55.4529439s) [noise=true , address=145.79.227.47:4265 , piece_id=JI4H7VUUERKAAX2Y6USZU2FTGSLU7GVPSNT73TXKIZZANBL5EDWA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.582139415 (613h12m55.45322246s) [noise=true , address=45.91.116.10:4081 , piece_id=XEGUJAS64MBGEYPSRMICVZ36R7OW4CGKHCYOXRFXIPGVFHHTG2FA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.582410030 (613h12m55.453493066s) [noise=true , address=145.79.227.41:4086 , piece_id=NDEO3MG5XY37OYTH6F6FNZGPENAAPIPF66FGETX6SSFTMEVWMMCQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.582531452 (613h12m55.453614487s) [noise=true , address=145.79.227.50:4373 , piece_id=R4HJ3LANTIYXX2PFW5YSPNAZU5RSGYOH536VN54IN4W2FJWKJZIA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.582736252 (613h12m55.453819287s) [noise=true , address=77.90.31.11:4556 , piece_id=3YEXD7HCKDVX4YIRGNZBYIU4ERD5W36DFCGZHESP3PRIGDPZP7TQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.582951672 (613h12m55.454034717s) [noise=true , address=145.79.227.25:4617 , piece_id=DIEL3BTDZIWUYVGYXLOSLJEEGGKRB7MKSVDYC3NHGCKWHIDZUDZQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.583092561 (613h12m55.454175626s) [noise=true , address=145.79.227.28:4726 , piece_id=ZYYILSQVLETAUKBJJY2S6AQ2IWG4CERFT3MWPEOUNV2SHYQ5AFAA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.583467485 (613h12m55.45455052s) [noise=true , address=145.79.227.19:4078 , piece_id=NLTLXPSPLV4TWE6KLGX5XSCK6HLQPM55Z2BWOIHMQZXXUEDII7RA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.583474358 (613h12m55.454557403s) [noise=true , address=77.90.31.13:4629 , piece_id=44B3JBKREOSLD57UQLATVRSWGJM7JG7I2OUQ3ZBZIEWO4C5SX6QQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.583773216 (613h12m55.454856251s) [noise=true , address=145.79.227.37:4566 , piece_id=JARGBZWEL4UD76EEISZEFMJ5SWY7RUAULFSWR7STTKM4RMW4FYHQ, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.584038291 (613h12m55.455121336s) [noise=true , address=145.79.227.48:4303 , piece_id=26ZXX6PUMFGVDOOGWP2IUMDTHIVPBHG4EVHYOH5IDVGSLPNTYEMA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.584120858 (613h12m55.455203903s) [noise=true , address=145.79.227.27:4691 , piece_id=7N4LYT4FTRXI6JV3NI3L65U6NUAUMXR2ZH5FZIMUXKBNRTNNIUEA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.584501463 (613h12m55.455584498s) [noise=true , address=45.91.116.5:4300 , piece_id=NOG7354BIZCG5JG6STFE2PAX7KNIOCZEFLQLDVISEHX24KKSBUAA, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:55.584856520 (613h12m55.455939555s) [noise=true , address=145.79.227.46:4603 , piece_id=BZUHK45GEASBFGGHSFAWFXN5VDF2VFPJ7DNPSZDH6FFKYSSUXR6A, seg_pos={0 0}, upload=537372]: Uploading piece... Dec 31 04:41:57.283478314 (613h12m57.154561349s) [upload=537372]: Next returned nil segment Dec 31 04:41:57.283498974 (613h12m57.154582009s) [upload=537372]: Waiting for error group managing segments... Dec 31 04:41:57.568671938 (613h12m57.439754973s) [noise=true , address=145.79.227.50:4373 , piece_id=R4HJ3LANTIYXX2PFW5YSPNAZU5RSGYOH536VN54IN4W2FJWKJZIA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.568784803 (613h12m57.439867838s) [noise=true , address=145.79.227.46:4603 , piece_id=BZUHK45GEASBFGGHSFAWFXN5VDF2VFPJ7DNPSZDH6FFKYSSUXR6A, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.569149196 (613h12m57.440232231s) [noise=true , address=145.79.227.19:4078 , piece_id=NLTLXPSPLV4TWE6KLGX5XSCK6HLQPM55Z2BWOIHMQZXXUEDII7RA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.577960875 (613h12m57.44904391s) [noise=true , address=145.79.227.37:4566 , piece_id=JARGBZWEL4UD76EEISZEFMJ5SWY7RUAULFSWR7STTKM4RMW4FYHQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.579194625 (613h12m57.45027766s) [noise=true , address=145.79.227.41:4086 , piece_id=NDEO3MG5XY37OYTH6F6FNZGPENAAPIPF66FGETX6SSFTMEVWMMCQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.585963744 (613h12m57.457046789s) [noise=true , address=145.79.227.28:4726 , piece_id=ZYYILSQVLETAUKBJJY2S6AQ2IWG4CERFT3MWPEOUNV2SHYQ5AFAA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.587889711 (613h12m57.458972756s) [noise=true , address=145.79.227.25:4617 , piece_id=DIEL3BTDZIWUYVGYXLOSLJEEGGKRB7MKSVDYC3NHGCKWHIDZUDZQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.600165486 (613h12m57.471248521s) [noise=true , address=145.79.227.48:4303 , piece_id=26ZXX6PUMFGVDOOGWP2IUMDTHIVPBHG4EVHYOH5IDVGSLPNTYEMA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.601065660 (613h12m57.472148705s) [noise=true , address=145.79.227.47:4265 , piece_id=JI4H7VUUERKAAX2Y6USZU2FTGSLU7GVPSNT73TXKIZZANBL5EDWA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.628124203 (613h12m57.499207248s) [noise=true , address=145.79.227.27:4691 , piece_id=7N4LYT4FTRXI6JV3NI3L65U6NUAUMXR2ZH5FZIMUXKBNRTNNIUEA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.818341791 (613h12m57.689424826s) [noise=true , address=77.90.31.13:4629 , piece_id=44B3JBKREOSLD57UQLATVRSWGJM7JG7I2OUQ3ZBZIEWO4C5SX6QQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:57.868995641 (613h12m57.740078676s) [noise=true , address=77.90.31.11:4556 , piece_id=3YEXD7HCKDVX4YIRGNZBYIU4ERD5W36DFCGZHESP3PRIGDPZP7TQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:58.089298215 (613h12m57.96038125s) [noise=true , address=45.91.116.6:4975 , piece_id=JAWGUNYPYTC2KXAJJTOKVTABC6E3QBZRJOCI3WOW3YQH5VYXTBVQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: <nil> Dec 31 04:41:58.089317181 (613h12m57.960400216s) [seg_pos={0 0}, upload=537372]: Segment reached optimal threshold of 13 pieces. Dec 31 04:41:58.089524216 (613h12m57.960607251s) [noise=true , address=45.91.116.10:4081 , piece_id=XEGUJAS64MBGEYPSRMICVZ36R7OW4CGKHCYOXRFXIPGVFHHTG2FA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 04:41:58.089531209 (613h12m57.960614254s) [noise=true , address=45.91.116.10:4081 , piece_id=XEGUJAS64MBGEYPSRMICVZ36R7OW4CGKHCYOXRFXIPGVFHHTG2FA, seg_pos={0 0}, upload=537372]: Long tail cancellation due to optimal threshold Dec 31 04:41:58.089694680 (613h12m57.960777715s) [noise=true , address=45.91.116.5:4300 , piece_id=NOG7354BIZCG5JG6STFE2PAX7KNIOCZEFLQLDVISEHX24KKSBUAA, seg_pos={0 0}, upload=537372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 31 04:41:58.089714317 (613h12m57.960797352s) [noise=true , address=45.91.116.5:4300 , piece_id=NOG7354BIZCG5JG6STFE2PAX7KNIOCZEFLQLDVISEHX24KKSBUAA, seg_pos={0 0}, upload=537372]: Long tail cancellation due to optimal threshold Dec 31 04:41:58.089755436 (613h12m57.960838471s) [noise=true , address=45.91.116.7:4009 , piece_id=P2KQXKZJDY43VTVO7KW4PFQBP3NEL42SLSLPKFAYAUF4XXZ73OLQ, seg_pos={0 0}, upload=537372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 31 04:41:58.089763531 (613h12m57.960846567s) [noise=true , address=45.91.116.7:4009 , piece_id=P2KQXKZJDY43VTVO7KW4PFQBP3NEL42SLSLPKFAYAUF4XXZ73OLQ, seg_pos={0 0}, upload=537372]: Long tail cancellation due to optimal threshold Dec 31 04:41:58.089867910 (613h12m57.960950955s) [seg_pos={0 0}, upload=537372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 04:41:58.089935529 (613h12m57.961018564s) [upload=537372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 04:41:58.089940118 (613h12m57.961023153s) [upload=537372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 04:41:58.388150457 (613h12m58.259233502s) [upload=537372]: Done uploading segments... Dec 31 04:41:58.388168983 (613h12m58.259252018s) [upload=537372]: Upload finished. err: <nil> Duration: 6.102323345s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)