Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.X] duration: 30.002825987s trace: https://jaeger.dev.storj.io/trace/1e74d94fdbc47cc1 FAIL (2025-12-27 03:46:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: DWKRRZGDLNKSMJZBBBT77ARZBKL4G7WOX7T4PH3ZS2GJAPV42G5A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: KJPWYB375LMAF676HA3ZSXAYECVGYKIKR3523SJAR5CXC2TKVZVA) context deadline exceeded ======LOGS====== 3:45:53.573904649 (516h16m53.444987694s) [upload=452993]: Starting upload Dec 27 03:45:53.574022874 (516h16m53.445105929s) [upload=452993]: Done starting upload Dec 27 03:45:53.574300102 (516h16m53.445383147s) [upload=452993]: Uploading segments... Dec 27 03:45:53.574354516 (516h16m53.445437571s) [upload=452993]: Got next segment. Inline: false Dec 27 03:45:53.574484043 (516h16m53.445567088s) [upload=452993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:45:53.574490946 (516h16m53.445573991s) [upload=452993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:45:53.820588774 (516h16m53.691671809s) [seg_pos={0 0}, upload=452993]: Begin upload segment... Dec 27 03:45:53.820652394 (516h16m53.691735439s) [seg_pos={0 0}, upload=452993]: Done begin upload segment. Dec 27 03:45:53.820690487 (516h16m53.691773532s) [noise=true , address=145.79.227.11:4730 , piece_id=H4YRQ2TCQV3ZLIPCWHYP2QYVWR7PCHYBYY57V5U7O2TJPQZ44VCA, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.820728440 (516h16m53.691811485s) [noise=true , address=45.91.116.6:4974 , piece_id=JIIXYUML62MLFX6ZFLAVQ43FCKL2GTZ5YSKSOKJ77JIPWEUBWAVA, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.821409496 (516h16m53.692492541s) [noise=true , address=45.91.116.4:4940 , piece_id=7NQLTBRB3KUPFIMJ3PE4IASB7KTBUBCQLBUUJU2XODHXPISVGULA, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.821991935 (516h16m53.69307498s) [noise=true , address=145.79.227.46:4601 , piece_id=NUPJ5QAIWJSL2LFKZXZMVNVR6LH46FXHS6TWDL34EYHSXFEKONXQ, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.822184833 (516h16m53.693267878s) [noise=true , address=145.79.227.52:4446 , piece_id=CPODBWSTBXVKU4U7LIYTLIVLZGL46NF6PPQFVCHR5RVT4CUNLAGA, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.823995902 (516h16m53.695078947s) [noise=true , address=145.79.227.47:4267 , piece_id=XZKRQEYREIYEM34VX2C2ZHUC2KUHR67DBHGKFD5EK7G3EFA6VIEQ, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.824125849 (516h16m53.695208884s) [noise=true , address=45.91.116.11:4120 , piece_id=4TA2N7EWWED75XMOFFDDB36JJ54IL45JMJUFH7TKENVJ2CT2657Q, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.824966009 (516h16m53.696049044s) [noise=true , address=45.91.116.12:4226 , piece_id=I643GZEHYQS6OGSORXQTF4AVZXQ45R4MUGODSQ3XP4B6LAVLID4Q, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.825250701 (516h16m53.696333736s) [noise=true , address=145.79.227.3:4149 , piece_id=K53B2KSBCEKBQAPYYFVLSL35FHC5CKXGNP4SPDHNLWULR27W2QJQ, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.826358871 (516h16m53.697441906s) [noise=true , address=77.90.31.21:4880 , piece_id=WGIMB4EMFS45CGG23R2Y3O32ILASVUGWKL2AVNP4WMGRXIC2JJDA, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.826662299 (516h16m53.697745344s) [noise=true , address=145.79.227.9:4365 , piece_id=ZIQ225LA4S2LN7A2VTGWJ4AU3QAHN574ZMFYFFLIGHPQBDILB4OA, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.828057436 (516h16m53.699140471s) [noise=true , address=145.79.227.48:4303 , piece_id=PVUSYBYOHTQTWW4IZJ2PGTOJEN4T64JUPA6T6TGJK7APHQAFONTA, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.828078706 (516h16m53.699161741s) [noise=true , address=145.79.227.38:4977 , piece_id=XK52EZ2T54Q77FX3CHCGFSC2A47GR2PWLLMPE7J5TFTCFPV7S3LQ, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.829250588 (516h16m53.700333633s) [noise=true , address=145.79.227.42:4121 , piece_id=DEPMXT4HYCGJQGG6DIJRB3OUMKPD37G342QJCVVURSZJOSMJDNUA, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.829404019 (516h16m53.700487064s) [noise=true , address=45.91.116.5:4298 , piece_id=DWKRRZGDLNKSMJZBBBT77ARZBKL4G7WOX7T4PH3ZS2GJAPV42G5A, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:53.829865148 (516h16m53.700948183s) [noise=true , address=45.91.116.10:4082 , piece_id=KJPWYB375LMAF676HA3ZSXAYECVGYKIKR3523SJAR5CXC2TKVZVA, seg_pos={0 0}, upload=452993]: Uploading piece... Dec 27 03:45:55.547654550 (516h16m55.418737585s) [upload=452993]: Next returned nil segment Dec 27 03:45:55.547667595 (516h16m55.41875063s) [upload=452993]: Waiting for error group managing segments... Dec 27 03:45:55.834248771 (516h16m55.705331816s) [noise=true , address=145.79.227.9:4365 , piece_id=ZIQ225LA4S2LN7A2VTGWJ4AU3QAHN574ZMFYFFLIGHPQBDILB4OA, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:55.835625312 (516h16m55.706708357s) [noise=true , address=145.79.227.52:4446 , piece_id=CPODBWSTBXVKU4U7LIYTLIVLZGL46NF6PPQFVCHR5RVT4CUNLAGA, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:55.845583965 (516h16m55.71666701s) [noise=true , address=145.79.227.46:4601 , piece_id=NUPJ5QAIWJSL2LFKZXZMVNVR6LH46FXHS6TWDL34EYHSXFEKONXQ, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:55.848169609 (516h16m55.719252644s) [noise=true , address=145.79.227.38:4977 , piece_id=XK52EZ2T54Q77FX3CHCGFSC2A47GR2PWLLMPE7J5TFTCFPV7S3LQ, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:55.848549602 (516h16m55.719632638s) [noise=true , address=145.79.227.48:4303 , piece_id=PVUSYBYOHTQTWW4IZJ2PGTOJEN4T64JUPA6T6TGJK7APHQAFONTA, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:55.856419127 (516h16m55.727502162s) [noise=true , address=145.79.227.42:4121 , piece_id=DEPMXT4HYCGJQGG6DIJRB3OUMKPD37G342QJCVVURSZJOSMJDNUA, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:55.857651764 (516h16m55.728734809s) [noise=true , address=145.79.227.11:4730 , piece_id=H4YRQ2TCQV3ZLIPCWHYP2QYVWR7PCHYBYY57V5U7O2TJPQZ44VCA, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:55.870759654 (516h16m55.741842689s) [noise=true , address=145.79.227.47:4267 , piece_id=XZKRQEYREIYEM34VX2C2ZHUC2KUHR67DBHGKFD5EK7G3EFA6VIEQ, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:56.042983422 (516h16m55.914066467s) [noise=true , address=45.91.116.5:4298 , piece_id=DWKRRZGDLNKSMJZBBBT77ARZBKL4G7WOX7T4PH3ZS2GJAPV42G5A, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:56.060660322 (516h16m55.931743367s) [noise=true , address=45.91.116.4:4940 , piece_id=7NQLTBRB3KUPFIMJ3PE4IASB7KTBUBCQLBUUJU2XODHXPISVGULA, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:56.078627625 (516h16m55.94971066s) [noise=true , address=45.91.116.11:4120 , piece_id=4TA2N7EWWED75XMOFFDDB36JJ54IL45JMJUFH7TKENVJ2CT2657Q, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:56.099635268 (516h16m55.970718303s) [noise=true , address=45.91.116.10:4082 , piece_id=KJPWYB375LMAF676HA3ZSXAYECVGYKIKR3523SJAR5CXC2TKVZVA, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:56.119133456 (516h16m55.990216521s) [noise=true , address=77.90.31.21:4880 , piece_id=WGIMB4EMFS45CGG23R2Y3O32ILASVUGWKL2AVNP4WMGRXIC2JJDA, seg_pos={0 0}, upload=452993]: Done uploading piece. err: <nil> Dec 27 03:45:56.119163714 (516h16m55.990246749s) [seg_pos={0 0}, upload=452993]: Segment reached optimal threshold of 13 pieces. Dec 27 03:45:56.119421234 (516h16m55.990504289s) [noise=true , address=45.91.116.12:4226 , piece_id=I643GZEHYQS6OGSORXQTF4AVZXQ45R4MUGODSQ3XP4B6LAVLID4Q, seg_pos={0 0}, upload=452993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 03:45:56.119431684 (516h16m55.990514719s) [noise=true , address=45.91.116.12:4226 , piece_id=I643GZEHYQS6OGSORXQTF4AVZXQ45R4MUGODSQ3XP4B6LAVLID4Q, seg_pos={0 0}, upload=452993]: Long tail cancellation due to optimal threshold Dec 27 03:45:56.119689124 (516h16m55.990772159s) [noise=true , address=145.79.227.3:4149 , piece_id=K53B2KSBCEKBQAPYYFVLSL35FHC5CKXGNP4SPDHNLWULR27W2QJQ, seg_pos={0 0}, upload=452993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: write tcp 188.166.225.170:44906->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:44906->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44906->145.79.227.3:4149: use of closed network connection Dec 27 03:45:56.119706217 (516h16m55.990789262s) [noise=true , address=145.79.227.3:4149 , piece_id=K53B2KSBCEKBQAPYYFVLSL35FHC5CKXGNP4SPDHNLWULR27W2QJQ, seg_pos={0 0}, upload=452993]: Long tail cancellation due to optimal threshold Dec 27 03:45:56.119779366 (516h16m55.990862421s) [noise=true , address=45.91.116.6:4974 , piece_id=JIIXYUML62MLFX6ZFLAVQ43FCKL2GTZ5YSKSOKJ77JIPWEUBWAVA, seg_pos={0 0}, upload=452993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 03:45:56.119791008 (516h16m55.990874053s) [noise=true , address=45.91.116.6:4974 , piece_id=JIIXYUML62MLFX6ZFLAVQ43FCKL2GTZ5YSKSOKJ77JIPWEUBWAVA, seg_pos={0 0}, upload=452993]: Long tail cancellation due to optimal threshold Dec 27 03:45:56.119882312 (516h16m55.990965347s) [seg_pos={0 0}, upload=452993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:45:56.120030444 (516h16m55.991113489s) [upload=452993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:45:56.120040102 (516h16m55.991123137s) [upload=452993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:45:56.413888676 (516h16m56.284971711s) [upload=452993]: Done uploading segments... Dec 27 03:45:56.413908464 (516h16m56.284991499s) [upload=452993]: Upload finished. err: <nil> Duration: 30.002852628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.....X.] duration: 46.609419857s trace: https://jaeger.dev.storj.io/trace/49a234f930fe9aa0 FAIL (2025-12-27 03:46:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, Piece ID: HDQUHKGFAHG53HAXFJDLX773LEMWCQYKNEM4BJ6VZ2DULMXXBS7A) context deadline exceeded; piecestore: (Node ID: 1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, Piece ID: MAWWTC7PNDTOWQW54OTW25OQEK26M4H7TRQ4A3E5HTRYAQIUU6IA) context deadline exceeded; piecestore: (Node ID: 1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, Piece ID: JMSNIGJZHKBACYPPLMZHQQBDGGBLAGEU2XHPGUHRFXET3IEVTKGA) context deadline exceeded; piecestore: (Node ID: 1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, Piece ID: HQJP7YQBNYF4OP736TYERPM26FY3BYS4WPCU7OLXTPY7PLZZZUCA) context deadline exceeded ======LOGS====== , piece_id=HDQUHKGFAHG53HAXFJDLX773LEMWCQYKNEM4BJ6VZ2DULMXXBS7A, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.933338072 (516h15m21.804421118s) [noise=true , address=77.90.31.15:4702 , piece_id=JMSNIGJZHKBACYPPLMZHQQBDGGBLAGEU2XHPGUHRFXET3IEVTKGA, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.933764746 (516h15m21.804847791s) [noise=true , address=145.79.227.22:4544 , piece_id=JF247YITQMQC3NZLEGBKIIO6QD72CXNKUSYTQUVNR3XXY7UUFBFQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.934231624 (516h15m21.805314679s) [noise=true , address=77.90.31.21:4891 , piece_id=VM4U7WSL5RFYLFJYDCDDQHZJDE5WRSVVAGD2CKNKTRT56IR2H4XQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.934798012 (516h15m21.805881048s) [noise=true , address=77.90.31.21:4896 , piece_id=SRRVOZTNZ76HQ3SY3SS5W3SB7PAU744OZ6KKCHYDVIFDR4WZM5FQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.934861574 (516h15m21.805944609s) [noise=true , address=77.90.31.11:4562 , piece_id=MAWWTC7PNDTOWQW54OTW25OQEK26M4H7TRQ4A3E5HTRYAQIUU6IA, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.935217591 (516h15m21.806300636s) [noise=true , address=64.49.13.139:4478 , piece_id=YKCZBL5S25IRWZJXU56M5FM466MSF4LK6M7KSBWG6MPQZAZHIUTA, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.935292454 (516h15m21.806375499s) [noise=true , address=77.90.31.14:4680 , piece_id=4QPA4R7UWZEMGWGHLREAKHYHGHDJC4HESVTI3FZ7YQHIB3PONMWQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.935637741 (516h15m21.806720786s) [noise=true , address=145.79.227.2:4147 , piece_id=BSGCRJZ2ZBER2WDSQ6WMYUEUO5THN3PGDSM4ATWLBPXXGQODTZ4Q, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.935794641 (516h15m21.806877686s) [noise=true , address=145.79.227.40:4071 , piece_id=YTYINPI25AYQUONCFRIRWXUDMZBSAV4DTLHEKLA4IOYRF5E6ENDQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.936201115 (516h15m21.80728416s) [noise=true , address=45.91.116.12:4257 , piece_id=WEW22VUSULP6KHF74AA362TENQB2VMRE4THMRMVJQPB3Q4GSRM3Q, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.936221774 (516h15m21.807304819s) [noise=true , address=145.79.227.12:4795 , piece_id=VGVXP7KI2Q3ERXVMZXMCCUCWF2UIKS46OALG6JF7M6W7AEMAVCCQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.936658676 (516h15m21.807741721s) [noise=true , address=145.79.227.28:4748 , piece_id=FSTI76ZWCITD25U6SXHUOGMHFX62CKCNJQVDZWA3BOZ6SYT5UNKQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.937069989 (516h15m21.808153035s) [noise=true , address=64.49.13.138:4483 , piece_id=GPCVPEGZUAPDHZ37JGB447PODZ7RWXPHQ6WOJKESKCASQPXNJ3ZQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.937516038 (516h15m21.808599073s) [noise=true , address=77.90.31.17:4794 , piece_id=HQJP7YQBNYF4OP736TYERPM26FY3BYS4WPCU7OLXTPY7PLZZZUCA, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:23.223462066 (516h15m23.094545101s) [upload=452973]: Next returned nil segment Dec 27 03:44:23.223475331 (516h15m23.094558366s) [upload=452973]: Waiting for error group managing segments... Dec 27 03:44:23.587390936 (516h15m23.458473991s) [noise=true , address=64.49.13.138:4483 , piece_id=GPCVPEGZUAPDHZ37JGB447PODZ7RWXPHQ6WOJKESKCASQPXNJ3ZQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:23.628641525 (516h15m23.49972457s) [noise=true , address=64.49.13.139:4478 , piece_id=YKCZBL5S25IRWZJXU56M5FM466MSF4LK6M7KSBWG6MPQZAZHIUTA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:23.972286381 (516h15m23.843369416s) [noise=true , address=77.90.31.21:4891 , piece_id=VM4U7WSL5RFYLFJYDCDDQHZJDE5WRSVVAGD2CKNKTRT56IR2H4XQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:23.976993525 (516h15m23.84807656s) [noise=true , address=77.90.31.14:4680 , piece_id=4QPA4R7UWZEMGWGHLREAKHYHGHDJC4HESVTI3FZ7YQHIB3PONMWQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:24.081443797 (516h15m23.952526842s) [noise=true , address=77.90.31.17:4794 , piece_id=HQJP7YQBNYF4OP736TYERPM26FY3BYS4WPCU7OLXTPY7PLZZZUCA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:24.106761978 (516h15m23.977845013s) [noise=true , address=45.91.116.12:4257 , piece_id=WEW22VUSULP6KHF74AA362TENQB2VMRE4THMRMVJQPB3Q4GSRM3Q, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:24.157644298 (516h15m24.028727333s) [noise=true , address=45.91.116.13:4168 , piece_id=ODRFJXWG6KJRP3O6G5NE3NLXAKMUIXUSYMR4E3R7BOVJULDEFCLA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:24.386347111 (516h15m24.257430146s) [noise=true , address=5.231.73.3:4022 , piece_id=HDQUHKGFAHG53HAXFJDLX773LEMWCQYKNEM4BJ6VZ2DULMXXBS7A, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:24.525289173 (516h15m24.396372208s) [noise=true , address=45.91.116.11:4124 , piece_id=NKLLTZFXBD4LI36YK7OAROTRGOCTHE6CBPBGTAI2KMNGX6XCKMZQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:25.576378249 (516h15m25.447461284s) [noise=true , address=77.90.31.21:4896 , piece_id=SRRVOZTNZ76HQ3SY3SS5W3SB7PAU744OZ6KKCHYDVIFDR4WZM5FQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:25.577119550 (516h15m25.448202595s) [noise=true , address=77.90.31.15:4702 , piece_id=JMSNIGJZHKBACYPPLMZHQQBDGGBLAGEU2XHPGUHRFXET3IEVTKGA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:25.978033217 (516h15m25.849116292s) [noise=true , address=77.90.31.11:4562 , piece_id=MAWWTC7PNDTOWQW54OTW25OQEK26M4H7TRQ4A3E5HTRYAQIUU6IA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:41.750545384 (516h15m41.621628419s) [noise=true , address=145.79.227.26:4662 , piece_id=7DENGHPDS4DMQZNBFGPUGUGMXTN7WRA2BHQ5TS3ZNQA6TGO3XFMA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:42.188278671 (516h15m42.059361707s) [noise=true , address=145.79.227.12:4795 , piece_id=VGVXP7KI2Q3ERXVMZXMCCUCWF2UIKS46OALG6JF7M6W7AEMAVCCQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:42.188298810 (516h15m42.059381845s) [seg_pos={0 0}, upload=452973]: Segment reached optimal threshold of 14 pieces. Dec 27 03:44:42.188469876 (516h15m42.059552911s) [noise=true , address=145.79.227.22:4544 , piece_id=JF247YITQMQC3NZLEGBKIIO6QD72CXNKUSYTQUVNR3XXY7UUFBFQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YkVN8PYQuRCwyhAfuMPy9qZaA7ZqWqj6BTNjtUSVxNw6eq4Sx): protocol: expected piece hash; context canceled; EOF Dec 27 03:44:42.188476478 (516h15m42.059559523s) [noise=true , address=145.79.227.22:4544 , piece_id=JF247YITQMQC3NZLEGBKIIO6QD72CXNKUSYTQUVNR3XXY7UUFBFQ, seg_pos={0 0}, upload=452973]: Long tail cancellation due to optimal threshold Dec 27 03:44:42.188596648 (516h15m42.059679693s) [noise=true , address=145.79.227.28:4748 , piece_id=FSTI76ZWCITD25U6SXHUOGMHFX62CKCNJQVDZWA3BOZ6SYT5UNKQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aZ4HiQxxV3qYJLYViMi7me9mRnPPBgPihW8fwzeMTKXnwbUb9): protocol: write tcp 188.166.225.170:44078->145.79.227.28:4748: use of closed network connection; write tcp 188.166.225.170:44078->145.79.227.28:4748: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44078->145.79.227.28:4748: use of closed network connection Dec 27 03:44:42.188604012 (516h15m42.059687057s) [noise=true , address=145.79.227.28:4748 , piece_id=FSTI76ZWCITD25U6SXHUOGMHFX62CKCNJQVDZWA3BOZ6SYT5UNKQ, seg_pos={0 0}, upload=452973]: Long tail cancellation due to optimal threshold Dec 27 03:44:42.188611035 (516h15m42.05969407s) [noise=true , address=145.79.227.2:4147 , piece_id=BSGCRJZ2ZBER2WDSQ6WMYUEUO5THN3PGDSM4ATWLBPXXGQODTZ4Q, seg_pos={0 0}, upload=452973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nNFCM4wDSvGMeqF8MF6csgKDUV6QyozoC5fZwUaj7MN1Woz4P): protocol: write tcp 188.166.225.170:46484->145.79.227.2:4147: use of closed network connection; write tcp 188.166.225.170:46484->145.79.227.2:4147: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46484->145.79.227.2:4147: use of closed network connection Dec 27 03:44:42.188617196 (516h15m42.059700231s) [noise=true , address=145.79.227.2:4147 , piece_id=BSGCRJZ2ZBER2WDSQ6WMYUEUO5THN3PGDSM4ATWLBPXXGQODTZ4Q, seg_pos={0 0}, upload=452973]: Long tail cancellation due to optimal threshold Dec 27 03:44:42.188735723 (516h15m42.059818768s) [noise=true , address=145.79.227.40:4071 , piece_id=YTYINPI25AYQUONCFRIRWXUDMZBSAV4DTLHEKLA4IOYRF5E6ENDQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VqztUNDN3hhJAMvCJtPreW7BdtAWSofQxAtmDKfDiX5sNEfWY): protocol: expected piece hash; context canceled; EOF Dec 27 03:44:42.188743768 (516h15m42.059826813s) [noise=true , address=145.79.227.40:4071 , piece_id=YTYINPI25AYQUONCFRIRWXUDMZBSAV4DTLHEKLA4IOYRF5E6ENDQ, seg_pos={0 0}, upload=452973]: Long tail cancellation due to optimal threshold Dec 27 03:44:42.188813020 (516h15m42.059896065s) [seg_pos={0 0}, upload=452973]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:44:42.188959578 (516h15m42.060042623s) [upload=452973]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:44:42.188968445 (516h15m42.06005148s) [upload=452973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:44:42.591632905 (516h15m42.46271594s) [upload=452973]: Done uploading segments... Dec 27 03:44:42.591655288 (516h15m42.462738323s) [upload=452973]: Upload finished. err: <nil> Duration: 46.609459262s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [X...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX.] duration: 6.254354779s trace: https://jaeger.dev.storj.io/trace/7b2c99c34b14bf6a FAIL (2025-12-27 03:45:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 16h15m17.960038246s) [noise=true , address=145.79.227.52:4446 , piece_id=B4BFRG6H5VLTADDF4RPPF35VCJGXAWS2X3OM5URTQ2TDSUO2QPAA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.089039621 (516h15m17.960122666s) [noise=true , address=145.79.227.47:4267 , piece_id=T6F7NN4H27KJLTWN64GFESCJHSGY4KOKLVLJOIYWSFBDR6ARQR6A, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.089646396 (516h15m17.960729431s) [noise=true , address=145.79.227.15:4874 , piece_id=EENGVCXEHQXFGR257GZZ5UPNHQRGZX3D46L2QN4BLF36FO64IKNQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.089978308 (516h15m17.961061343s) [noise=true , address=145.79.227.42:4121 , piece_id=7FGGUOC4N65LFR7WAP4IDWIJPTPYZDG4XOZ6BSN72N6PBE7GDZQA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.090310892 (516h15m17.961393937s) [noise=true , address=145.79.227.3:4151 , piece_id=KTUQMYU2PGVXXIEMJ6YQG5GPTNBO6IIZDH2MNENLVQPV7N7ATMHQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.090722144 (516h15m17.961805189s) [noise=true , address=77.90.31.14:4663 , piece_id=DVQR5FVM2MCSAPY7RWBJDNKZYNT62HZVTDYK7XXLKAO2HTCO7WJA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.091006526 (516h15m17.962089571s) [noise=true , address=77.90.31.11:4554 , piece_id=NDWL2DMLK3FEJ5RFJZPP2EAVDZODRTIUEGUAQZPY5YZBDYPSMGTA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.091280447 (516h15m17.962363483s) [noise=true , address=45.91.116.10:4082 , piece_id=5T7FXDUGD3UUSQKRHKJ3BX64V2AF6Y3TUCGIGNWBCC5H5P7M5UWQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.092812074 (516h15m17.963895119s) [noise=true , address=45.91.116.8:4048 , piece_id=CR3GHVT6CPLUOKJKYPE3JPSUOEWNLPW57UA2FITRLS7JFDFYTMJQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.094107882 (516h15m17.965190917s) [noise=true , address=145.79.227.16:4911 , piece_id=BUO5MQ5SQ5ZUL3MHUEY4NBWPEYGDMRYTDAWM32R36UQS6AJGYJ6Q, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.095577761 (516h15m17.966660806s) [noise=true , address=145.79.227.45:4229 , piece_id=XBMDKXOHJK3XBZII2O3SY7QXH5TLHT26QYJBDYCARO57QA4UCV6Q, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.096827881 (516h15m17.967910916s) [noise=true , address=145.79.227.5:4222 , piece_id=4L4X5P2GNLNKFUVMTPK74QQU5XMISHWKRGDMJU7LVJ6ZYEM5P5RQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.098089644 (516h15m17.969172679s) [noise=true , address=77.90.31.21:4880 , piece_id=VECZ4P63IFR6E2YT2PWDWRPF5S3H6MO25OQKILRU3UHB3DAYVVTQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.099363430 (516h15m17.970446465s) [noise=true , address=145.79.227.11:4729 , piece_id=ZD6ECQVPW5WJVN6GQHDS7PWBVPP5GX77ZFICSRNBB7ICIGCCGIBA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.100505033 (516h15m17.971588068s) [noise=true , address=45.91.116.14:4192 , piece_id=ABUPV2GYD7ZHGLGJFNWS6FGHNYHGJLFZ7X6YY7G5IDPALSVEFYMA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.101611931 (516h15m17.972694976s) [noise=true , address=145.79.227.46:4602 , piece_id=NETN672RCGCQWGZNT4EM3BYAYR3MXKLXMRIG4EPQ5CDI2XAHELJA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:19.938740457 (516h15m19.809823502s) [upload=452972]: Next returned nil segment Dec 27 03:44:19.938757599 (516h15m19.809840654s) [upload=452972]: Waiting for error group managing segments... Dec 27 03:44:20.303442609 (516h15m20.174525644s) [noise=true , address=77.90.31.11:4554 , piece_id=NDWL2DMLK3FEJ5RFJZPP2EAVDZODRTIUEGUAQZPY5YZBDYPSMGTA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:20.305867036 (516h15m20.176950071s) [noise=true , address=77.90.31.21:4880 , piece_id=VECZ4P63IFR6E2YT2PWDWRPF5S3H6MO25OQKILRU3UHB3DAYVVTQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:20.481472803 (516h15m20.352555838s) [noise=true , address=77.90.31.14:4663 , piece_id=DVQR5FVM2MCSAPY7RWBJDNKZYNT62HZVTDYK7XXLKAO2HTCO7WJA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:20.717148073 (516h15m20.588231108s) [noise=true , address=45.91.116.10:4082 , piece_id=5T7FXDUGD3UUSQKRHKJ3BX64V2AF6Y3TUCGIGNWBCC5H5P7M5UWQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:21.456905454 (516h15m21.327988499s) [noise=true , address=45.91.116.14:4192 , piece_id=ABUPV2GYD7ZHGLGJFNWS6FGHNYHGJLFZ7X6YY7G5IDPALSVEFYMA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:22.729539964 (516h15m22.600623009s) [noise=true , address=45.91.116.8:4048 , piece_id=CR3GHVT6CPLUOKJKYPE3JPSUOEWNLPW57UA2FITRLS7JFDFYTMJQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:40.676989347 (516h15m40.548072382s) [noise=true , address=145.79.227.16:4911 , piece_id=BUO5MQ5SQ5ZUL3MHUEY4NBWPEYGDMRYTDAWM32R36UQS6AJGYJ6Q, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:41.953873210 (516h15m41.824956255s) [noise=true , address=145.79.227.15:4874 , piece_id=EENGVCXEHQXFGR257GZZ5UPNHQRGZX3D46L2QN4BLF36FO64IKNQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:46.578593515 (516h15m46.44967655s) [noise=true , address=145.79.227.52:4446 , piece_id=B4BFRG6H5VLTADDF4RPPF35VCJGXAWS2X3OM5URTQ2TDSUO2QPAA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:47.165218563 (516h15m47.036301618s) [noise=true , address=145.79.227.3:4151 , piece_id=KTUQMYU2PGVXXIEMJ6YQG5GPTNBO6IIZDH2MNENLVQPV7N7ATMHQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:41132->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:41132->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41132->145.79.227.3:4151: use of closed network connection Dec 27 03:44:47.165295258 (516h15m47.036378293s) [noise=true , address=145.79.227.5:4222 , piece_id=4L4X5P2GNLNKFUVMTPK74QQU5XMISHWKRGDMJU7LVJ6ZYEM5P5RQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:52030->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:52030->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52030->145.79.227.5:4222: use of closed network connection Dec 27 03:44:47.165326127 (516h15m47.036409162s) [noise=true , address=145.79.227.46:4602 , piece_id=NETN672RCGCQWGZNT4EM3BYAYR3MXKLXMRIG4EPQ5CDI2XAHELJA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d, address:145.79.227.46:4602): protocol: write tcp 188.166.225.170:49364->145.79.227.46:4602: use of closed network connection; write tcp 188.166.225.170:49364->145.79.227.46:4602: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49364->145.79.227.46:4602: use of closed network connection Dec 27 03:44:47.165401861 (516h15m47.036484896s) [noise=true , address=145.79.227.47:4267 , piece_id=T6F7NN4H27KJLTWN64GFESCJHSGY4KOKLVLJOIYWSFBDR6ARQR6A, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:44836->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:44836->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44836->145.79.227.47:4267: use of closed network connection Dec 27 03:44:47.165425917 (516h15m47.036508962s) [noise=true , address=145.79.227.11:4729 , piece_id=ZD6ECQVPW5WJVN6GQHDS7PWBVPP5GX77ZFICSRNBB7ICIGCCGIBA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: expected piece hash; context deadline exceeded; EOF Dec 27 03:44:47.165498486 (516h15m47.036581521s) [noise=true , address=145.79.227.42:4121 , piece_id=7FGGUOC4N65LFR7WAP4IDWIJPTPYZDG4XOZ6BSN72N6PBE7GDZQA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:51342->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.42:4121: use of closed network connection Dec 27 03:44:47.165540676 (516h15m47.036623711s) [noise=true , address=145.79.227.45:4229 , piece_id=XBMDKXOHJK3XBZII2O3SY7QXH5TLHT26QYJBDYCARO57QA4UCV6Q, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:49238->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:49238->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49238->145.79.227.45:4229: use of closed network connection Dec 27 03:44:47.165566024 (516h15m47.036649059s) [seg_pos={0 0}, upload=452972]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 27 03:44:47.399864223 (516h15m47.270947268s) [upload=452972]: Done uploading segments... Dec 27 03:44:47.399901143 (516h15m47.270984178s) [upload=452972]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.254380819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.....X] duration: 1m0.003159599s trace: https://jaeger.dev.storj.io/trace/49a234f930fe9aa0 FAIL (2025-12-27 03:45:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B, Piece ID: HDQUHKGFAHG53HAXFJDLX773LEMWCQYKNEM4BJ6VZ2DULMXXBS7A) context deadline exceeded; piecestore: (Node ID: 1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, Piece ID: MAWWTC7PNDTOWQW54OTW25OQEK26M4H7TRQ4A3E5HTRYAQIUU6IA) context deadline exceeded; piecestore: (Node ID: 1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, Piece ID: JMSNIGJZHKBACYPPLMZHQQBDGGBLAGEU2XHPGUHRFXET3IEVTKGA) context deadline exceeded; piecestore: (Node ID: 1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, Piece ID: HQJP7YQBNYF4OP736TYERPM26FY3BYS4WPCU7OLXTPY7PLZZZUCA) context deadline exceeded ======LOGS====== , piece_id=HDQUHKGFAHG53HAXFJDLX773LEMWCQYKNEM4BJ6VZ2DULMXXBS7A, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.933338072 (516h15m21.804421118s) [noise=true , address=77.90.31.15:4702 , piece_id=JMSNIGJZHKBACYPPLMZHQQBDGGBLAGEU2XHPGUHRFXET3IEVTKGA, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.933764746 (516h15m21.804847791s) [noise=true , address=145.79.227.22:4544 , piece_id=JF247YITQMQC3NZLEGBKIIO6QD72CXNKUSYTQUVNR3XXY7UUFBFQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.934231624 (516h15m21.805314679s) [noise=true , address=77.90.31.21:4891 , piece_id=VM4U7WSL5RFYLFJYDCDDQHZJDE5WRSVVAGD2CKNKTRT56IR2H4XQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.934798012 (516h15m21.805881048s) [noise=true , address=77.90.31.21:4896 , piece_id=SRRVOZTNZ76HQ3SY3SS5W3SB7PAU744OZ6KKCHYDVIFDR4WZM5FQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.934861574 (516h15m21.805944609s) [noise=true , address=77.90.31.11:4562 , piece_id=MAWWTC7PNDTOWQW54OTW25OQEK26M4H7TRQ4A3E5HTRYAQIUU6IA, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.935217591 (516h15m21.806300636s) [noise=true , address=64.49.13.139:4478 , piece_id=YKCZBL5S25IRWZJXU56M5FM466MSF4LK6M7KSBWG6MPQZAZHIUTA, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.935292454 (516h15m21.806375499s) [noise=true , address=77.90.31.14:4680 , piece_id=4QPA4R7UWZEMGWGHLREAKHYHGHDJC4HESVTI3FZ7YQHIB3PONMWQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.935637741 (516h15m21.806720786s) [noise=true , address=145.79.227.2:4147 , piece_id=BSGCRJZ2ZBER2WDSQ6WMYUEUO5THN3PGDSM4ATWLBPXXGQODTZ4Q, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.935794641 (516h15m21.806877686s) [noise=true , address=145.79.227.40:4071 , piece_id=YTYINPI25AYQUONCFRIRWXUDMZBSAV4DTLHEKLA4IOYRF5E6ENDQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.936201115 (516h15m21.80728416s) [noise=true , address=45.91.116.12:4257 , piece_id=WEW22VUSULP6KHF74AA362TENQB2VMRE4THMRMVJQPB3Q4GSRM3Q, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.936221774 (516h15m21.807304819s) [noise=true , address=145.79.227.12:4795 , piece_id=VGVXP7KI2Q3ERXVMZXMCCUCWF2UIKS46OALG6JF7M6W7AEMAVCCQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.936658676 (516h15m21.807741721s) [noise=true , address=145.79.227.28:4748 , piece_id=FSTI76ZWCITD25U6SXHUOGMHFX62CKCNJQVDZWA3BOZ6SYT5UNKQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.937069989 (516h15m21.808153035s) [noise=true , address=64.49.13.138:4483 , piece_id=GPCVPEGZUAPDHZ37JGB447PODZ7RWXPHQ6WOJKESKCASQPXNJ3ZQ, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:21.937516038 (516h15m21.808599073s) [noise=true , address=77.90.31.17:4794 , piece_id=HQJP7YQBNYF4OP736TYERPM26FY3BYS4WPCU7OLXTPY7PLZZZUCA, seg_pos={0 0}, upload=452973]: Uploading piece... Dec 27 03:44:23.223462066 (516h15m23.094545101s) [upload=452973]: Next returned nil segment Dec 27 03:44:23.223475331 (516h15m23.094558366s) [upload=452973]: Waiting for error group managing segments... Dec 27 03:44:23.587390936 (516h15m23.458473991s) [noise=true , address=64.49.13.138:4483 , piece_id=GPCVPEGZUAPDHZ37JGB447PODZ7RWXPHQ6WOJKESKCASQPXNJ3ZQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:23.628641525 (516h15m23.49972457s) [noise=true , address=64.49.13.139:4478 , piece_id=YKCZBL5S25IRWZJXU56M5FM466MSF4LK6M7KSBWG6MPQZAZHIUTA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:23.972286381 (516h15m23.843369416s) [noise=true , address=77.90.31.21:4891 , piece_id=VM4U7WSL5RFYLFJYDCDDQHZJDE5WRSVVAGD2CKNKTRT56IR2H4XQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:23.976993525 (516h15m23.84807656s) [noise=true , address=77.90.31.14:4680 , piece_id=4QPA4R7UWZEMGWGHLREAKHYHGHDJC4HESVTI3FZ7YQHIB3PONMWQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:24.081443797 (516h15m23.952526842s) [noise=true , address=77.90.31.17:4794 , piece_id=HQJP7YQBNYF4OP736TYERPM26FY3BYS4WPCU7OLXTPY7PLZZZUCA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:24.106761978 (516h15m23.977845013s) [noise=true , address=45.91.116.12:4257 , piece_id=WEW22VUSULP6KHF74AA362TENQB2VMRE4THMRMVJQPB3Q4GSRM3Q, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:24.157644298 (516h15m24.028727333s) [noise=true , address=45.91.116.13:4168 , piece_id=ODRFJXWG6KJRP3O6G5NE3NLXAKMUIXUSYMR4E3R7BOVJULDEFCLA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:24.386347111 (516h15m24.257430146s) [noise=true , address=5.231.73.3:4022 , piece_id=HDQUHKGFAHG53HAXFJDLX773LEMWCQYKNEM4BJ6VZ2DULMXXBS7A, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:24.525289173 (516h15m24.396372208s) [noise=true , address=45.91.116.11:4124 , piece_id=NKLLTZFXBD4LI36YK7OAROTRGOCTHE6CBPBGTAI2KMNGX6XCKMZQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:25.576378249 (516h15m25.447461284s) [noise=true , address=77.90.31.21:4896 , piece_id=SRRVOZTNZ76HQ3SY3SS5W3SB7PAU744OZ6KKCHYDVIFDR4WZM5FQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:25.577119550 (516h15m25.448202595s) [noise=true , address=77.90.31.15:4702 , piece_id=JMSNIGJZHKBACYPPLMZHQQBDGGBLAGEU2XHPGUHRFXET3IEVTKGA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:25.978033217 (516h15m25.849116292s) [noise=true , address=77.90.31.11:4562 , piece_id=MAWWTC7PNDTOWQW54OTW25OQEK26M4H7TRQ4A3E5HTRYAQIUU6IA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:41.750545384 (516h15m41.621628419s) [noise=true , address=145.79.227.26:4662 , piece_id=7DENGHPDS4DMQZNBFGPUGUGMXTN7WRA2BHQ5TS3ZNQA6TGO3XFMA, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:42.188278671 (516h15m42.059361707s) [noise=true , address=145.79.227.12:4795 , piece_id=VGVXP7KI2Q3ERXVMZXMCCUCWF2UIKS46OALG6JF7M6W7AEMAVCCQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: <nil> Dec 27 03:44:42.188298810 (516h15m42.059381845s) [seg_pos={0 0}, upload=452973]: Segment reached optimal threshold of 14 pieces. Dec 27 03:44:42.188469876 (516h15m42.059552911s) [noise=true , address=145.79.227.22:4544 , piece_id=JF247YITQMQC3NZLEGBKIIO6QD72CXNKUSYTQUVNR3XXY7UUFBFQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YkVN8PYQuRCwyhAfuMPy9qZaA7ZqWqj6BTNjtUSVxNw6eq4Sx): protocol: expected piece hash; context canceled; EOF Dec 27 03:44:42.188476478 (516h15m42.059559523s) [noise=true , address=145.79.227.22:4544 , piece_id=JF247YITQMQC3NZLEGBKIIO6QD72CXNKUSYTQUVNR3XXY7UUFBFQ, seg_pos={0 0}, upload=452973]: Long tail cancellation due to optimal threshold Dec 27 03:44:42.188596648 (516h15m42.059679693s) [noise=true , address=145.79.227.28:4748 , piece_id=FSTI76ZWCITD25U6SXHUOGMHFX62CKCNJQVDZWA3BOZ6SYT5UNKQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aZ4HiQxxV3qYJLYViMi7me9mRnPPBgPihW8fwzeMTKXnwbUb9): protocol: write tcp 188.166.225.170:44078->145.79.227.28:4748: use of closed network connection; write tcp 188.166.225.170:44078->145.79.227.28:4748: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44078->145.79.227.28:4748: use of closed network connection Dec 27 03:44:42.188604012 (516h15m42.059687057s) [noise=true , address=145.79.227.28:4748 , piece_id=FSTI76ZWCITD25U6SXHUOGMHFX62CKCNJQVDZWA3BOZ6SYT5UNKQ, seg_pos={0 0}, upload=452973]: Long tail cancellation due to optimal threshold Dec 27 03:44:42.188611035 (516h15m42.05969407s) [noise=true , address=145.79.227.2:4147 , piece_id=BSGCRJZ2ZBER2WDSQ6WMYUEUO5THN3PGDSM4ATWLBPXXGQODTZ4Q, seg_pos={0 0}, upload=452973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nNFCM4wDSvGMeqF8MF6csgKDUV6QyozoC5fZwUaj7MN1Woz4P): protocol: write tcp 188.166.225.170:46484->145.79.227.2:4147: use of closed network connection; write tcp 188.166.225.170:46484->145.79.227.2:4147: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46484->145.79.227.2:4147: use of closed network connection Dec 27 03:44:42.188617196 (516h15m42.059700231s) [noise=true , address=145.79.227.2:4147 , piece_id=BSGCRJZ2ZBER2WDSQ6WMYUEUO5THN3PGDSM4ATWLBPXXGQODTZ4Q, seg_pos={0 0}, upload=452973]: Long tail cancellation due to optimal threshold Dec 27 03:44:42.188735723 (516h15m42.059818768s) [noise=true , address=145.79.227.40:4071 , piece_id=YTYINPI25AYQUONCFRIRWXUDMZBSAV4DTLHEKLA4IOYRF5E6ENDQ, seg_pos={0 0}, upload=452973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VqztUNDN3hhJAMvCJtPreW7BdtAWSofQxAtmDKfDiX5sNEfWY): protocol: expected piece hash; context canceled; EOF Dec 27 03:44:42.188743768 (516h15m42.059826813s) [noise=true , address=145.79.227.40:4071 , piece_id=YTYINPI25AYQUONCFRIRWXUDMZBSAV4DTLHEKLA4IOYRF5E6ENDQ, seg_pos={0 0}, upload=452973]: Long tail cancellation due to optimal threshold Dec 27 03:44:42.188813020 (516h15m42.059896065s) [seg_pos={0 0}, upload=452973]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:44:42.188959578 (516h15m42.060042623s) [upload=452973]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:44:42.188968445 (516h15m42.06005148s) [upload=452973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:44:42.591632905 (516h15m42.46271594s) [upload=452973]: Done uploading segments... Dec 27 03:44:42.591655288 (516h15m42.462738323s) [upload=452973]: Upload finished. err: <nil> Duration: 1m0.003182413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.XX] duration: 30.236117156s trace: https://jaeger.dev.storj.io/trace/7b2c99c34b14bf6a FAIL (2025-12-27 03:44:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 16h15m17.960038246s) [noise=true , address=145.79.227.52:4446 , piece_id=B4BFRG6H5VLTADDF4RPPF35VCJGXAWS2X3OM5URTQ2TDSUO2QPAA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.089039621 (516h15m17.960122666s) [noise=true , address=145.79.227.47:4267 , piece_id=T6F7NN4H27KJLTWN64GFESCJHSGY4KOKLVLJOIYWSFBDR6ARQR6A, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.089646396 (516h15m17.960729431s) [noise=true , address=145.79.227.15:4874 , piece_id=EENGVCXEHQXFGR257GZZ5UPNHQRGZX3D46L2QN4BLF36FO64IKNQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.089978308 (516h15m17.961061343s) [noise=true , address=145.79.227.42:4121 , piece_id=7FGGUOC4N65LFR7WAP4IDWIJPTPYZDG4XOZ6BSN72N6PBE7GDZQA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.090310892 (516h15m17.961393937s) [noise=true , address=145.79.227.3:4151 , piece_id=KTUQMYU2PGVXXIEMJ6YQG5GPTNBO6IIZDH2MNENLVQPV7N7ATMHQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.090722144 (516h15m17.961805189s) [noise=true , address=77.90.31.14:4663 , piece_id=DVQR5FVM2MCSAPY7RWBJDNKZYNT62HZVTDYK7XXLKAO2HTCO7WJA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.091006526 (516h15m17.962089571s) [noise=true , address=77.90.31.11:4554 , piece_id=NDWL2DMLK3FEJ5RFJZPP2EAVDZODRTIUEGUAQZPY5YZBDYPSMGTA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.091280447 (516h15m17.962363483s) [noise=true , address=45.91.116.10:4082 , piece_id=5T7FXDUGD3UUSQKRHKJ3BX64V2AF6Y3TUCGIGNWBCC5H5P7M5UWQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.092812074 (516h15m17.963895119s) [noise=true , address=45.91.116.8:4048 , piece_id=CR3GHVT6CPLUOKJKYPE3JPSUOEWNLPW57UA2FITRLS7JFDFYTMJQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.094107882 (516h15m17.965190917s) [noise=true , address=145.79.227.16:4911 , piece_id=BUO5MQ5SQ5ZUL3MHUEY4NBWPEYGDMRYTDAWM32R36UQS6AJGYJ6Q, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.095577761 (516h15m17.966660806s) [noise=true , address=145.79.227.45:4229 , piece_id=XBMDKXOHJK3XBZII2O3SY7QXH5TLHT26QYJBDYCARO57QA4UCV6Q, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.096827881 (516h15m17.967910916s) [noise=true , address=145.79.227.5:4222 , piece_id=4L4X5P2GNLNKFUVMTPK74QQU5XMISHWKRGDMJU7LVJ6ZYEM5P5RQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.098089644 (516h15m17.969172679s) [noise=true , address=77.90.31.21:4880 , piece_id=VECZ4P63IFR6E2YT2PWDWRPF5S3H6MO25OQKILRU3UHB3DAYVVTQ, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.099363430 (516h15m17.970446465s) [noise=true , address=145.79.227.11:4729 , piece_id=ZD6ECQVPW5WJVN6GQHDS7PWBVPP5GX77ZFICSRNBB7ICIGCCGIBA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.100505033 (516h15m17.971588068s) [noise=true , address=45.91.116.14:4192 , piece_id=ABUPV2GYD7ZHGLGJFNWS6FGHNYHGJLFZ7X6YY7G5IDPALSVEFYMA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:18.101611931 (516h15m17.972694976s) [noise=true , address=145.79.227.46:4602 , piece_id=NETN672RCGCQWGZNT4EM3BYAYR3MXKLXMRIG4EPQ5CDI2XAHELJA, seg_pos={0 0}, upload=452972]: Uploading piece... Dec 27 03:44:19.938740457 (516h15m19.809823502s) [upload=452972]: Next returned nil segment Dec 27 03:44:19.938757599 (516h15m19.809840654s) [upload=452972]: Waiting for error group managing segments... Dec 27 03:44:20.303442609 (516h15m20.174525644s) [noise=true , address=77.90.31.11:4554 , piece_id=NDWL2DMLK3FEJ5RFJZPP2EAVDZODRTIUEGUAQZPY5YZBDYPSMGTA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:20.305867036 (516h15m20.176950071s) [noise=true , address=77.90.31.21:4880 , piece_id=VECZ4P63IFR6E2YT2PWDWRPF5S3H6MO25OQKILRU3UHB3DAYVVTQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:20.481472803 (516h15m20.352555838s) [noise=true , address=77.90.31.14:4663 , piece_id=DVQR5FVM2MCSAPY7RWBJDNKZYNT62HZVTDYK7XXLKAO2HTCO7WJA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:20.717148073 (516h15m20.588231108s) [noise=true , address=45.91.116.10:4082 , piece_id=5T7FXDUGD3UUSQKRHKJ3BX64V2AF6Y3TUCGIGNWBCC5H5P7M5UWQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:21.456905454 (516h15m21.327988499s) [noise=true , address=45.91.116.14:4192 , piece_id=ABUPV2GYD7ZHGLGJFNWS6FGHNYHGJLFZ7X6YY7G5IDPALSVEFYMA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:22.729539964 (516h15m22.600623009s) [noise=true , address=45.91.116.8:4048 , piece_id=CR3GHVT6CPLUOKJKYPE3JPSUOEWNLPW57UA2FITRLS7JFDFYTMJQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:40.676989347 (516h15m40.548072382s) [noise=true , address=145.79.227.16:4911 , piece_id=BUO5MQ5SQ5ZUL3MHUEY4NBWPEYGDMRYTDAWM32R36UQS6AJGYJ6Q, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:41.953873210 (516h15m41.824956255s) [noise=true , address=145.79.227.15:4874 , piece_id=EENGVCXEHQXFGR257GZZ5UPNHQRGZX3D46L2QN4BLF36FO64IKNQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:46.578593515 (516h15m46.44967655s) [noise=true , address=145.79.227.52:4446 , piece_id=B4BFRG6H5VLTADDF4RPPF35VCJGXAWS2X3OM5URTQ2TDSUO2QPAA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: <nil> Dec 27 03:44:47.165218563 (516h15m47.036301618s) [noise=true , address=145.79.227.3:4151 , piece_id=KTUQMYU2PGVXXIEMJ6YQG5GPTNBO6IIZDH2MNENLVQPV7N7ATMHQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: write tcp 188.166.225.170:41132->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:41132->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41132->145.79.227.3:4151: use of closed network connection Dec 27 03:44:47.165295258 (516h15m47.036378293s) [noise=true , address=145.79.227.5:4222 , piece_id=4L4X5P2GNLNKFUVMTPK74QQU5XMISHWKRGDMJU7LVJ6ZYEM5P5RQ, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:52030->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:52030->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52030->145.79.227.5:4222: use of closed network connection Dec 27 03:44:47.165326127 (516h15m47.036409162s) [noise=true , address=145.79.227.46:4602 , piece_id=NETN672RCGCQWGZNT4EM3BYAYR3MXKLXMRIG4EPQ5CDI2XAHELJA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d, address:145.79.227.46:4602): protocol: write tcp 188.166.225.170:49364->145.79.227.46:4602: use of closed network connection; write tcp 188.166.225.170:49364->145.79.227.46:4602: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49364->145.79.227.46:4602: use of closed network connection Dec 27 03:44:47.165401861 (516h15m47.036484896s) [noise=true , address=145.79.227.47:4267 , piece_id=T6F7NN4H27KJLTWN64GFESCJHSGY4KOKLVLJOIYWSFBDR6ARQR6A, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:44836->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:44836->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44836->145.79.227.47:4267: use of closed network connection Dec 27 03:44:47.165425917 (516h15m47.036508962s) [noise=true , address=145.79.227.11:4729 , piece_id=ZD6ECQVPW5WJVN6GQHDS7PWBVPP5GX77ZFICSRNBB7ICIGCCGIBA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: expected piece hash; context deadline exceeded; EOF Dec 27 03:44:47.165498486 (516h15m47.036581521s) [noise=true , address=145.79.227.42:4121 , piece_id=7FGGUOC4N65LFR7WAP4IDWIJPTPYZDG4XOZ6BSN72N6PBE7GDZQA, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:51342->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:51342->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51342->145.79.227.42:4121: use of closed network connection Dec 27 03:44:47.165540676 (516h15m47.036623711s) [noise=true , address=145.79.227.45:4229 , piece_id=XBMDKXOHJK3XBZII2O3SY7QXH5TLHT26QYJBDYCARO57QA4UCV6Q, seg_pos={0 0}, upload=452972]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:49238->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:49238->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49238->145.79.227.45:4229: use of closed network connection Dec 27 03:44:47.165566024 (516h15m47.036649059s) [seg_pos={0 0}, upload=452972]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 27 03:44:47.399864223 (516h15m47.270947268s) [upload=452972]: Done uploading segments... Dec 27 03:44:47.399901143 (516h15m47.270984178s) [upload=452972]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.236140811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 55.566767245s trace: https://jaeger.dev.storj.io/trace/2f1833ad3fc26997 FAIL (2025-12-27 03:44:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.046968512 (516h10m16.918051557s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047520183 (516h10m16.918603228s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047984678 (516h10m16.919067723s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048465613 (516h10m16.919548658s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048734075 (516h10m16.91981712s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049142162 (516h10m16.920225207s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049245980 (516h10m16.920329025s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049769577 (516h10m16.920852612s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050319855 (516h10m16.9214029s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050902745 (516h10m16.92198579s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051099419 (516h10m16.922182464s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051941683 (516h10m16.923024738s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.052719805 (516h10m16.92380284s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.054577933 (516h10m16.925660969s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.058395733 (516h10m16.929478778s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:18.699503155 (516h10m18.5705862s) [upload=452901]: Next returned nil segment Dec 27 03:39:18.699553180 (516h10m18.570636226s) [upload=452901]: Waiting for error group managing segments... Dec 27 03:39:19.134636692 (516h10m19.005719727s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.202132478 (516h10m19.073215523s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.203422765 (516h10m19.07450581s) [noise=true , address=45.91.116.14:4194 , piece_id=Z2K5GGTN367GZ6NLOKWTBTVD2HNYEFEAA2MDGSZ2YMZG3XAFGEZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.235358065 (516h10m19.106441101s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.243623684 (516h10m19.114706729s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.298044432 (516h10m19.169127477s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.523897079 (516h10m19.394980124s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:20.015944923 (516h10m19.887027968s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:21.684763342 (516h10m21.555846377s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:22.446094653 (516h10m22.317177698s) [noise=true , address=77.90.31.18:4813 , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:24.365452157 (516h10m24.236535202s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:31.513341529 (516h10m31.384424564s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:32.290856236 (516h10m32.161939301s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578538275 (516h10m35.44962131s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578557731 (516h10m35.449640766s) [seg_pos={0 0}, upload=452901]: Segment reached optimal threshold of 14 pieces. Dec 27 03:39:35.578728707 (516h10m35.449811742s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Dec 27 03:39:35.578734489 (516h10m35.449817524s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.578899161 (516h10m35.449982207s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo): protocol: rpc: dial tcp 145.79.227.29:4949: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.578904221 (516h10m35.449987266s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579127447 (516h10m35.450210482s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: read tcp 188.166.225.170:58082->145.79.227.25:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579135101 (516h10m35.450218136s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579158115 (516h10m35.45024115s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: read tcp 188.166.225.170:40188->145.79.227.40:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579162754 (516h10m35.450245799s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579173815 (516h10m35.45025686s) [seg_pos={0 0}, upload=452901]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:39:35.579230252 (516h10m35.450313287s) [upload=452901]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:39:35.579234711 (516h10m35.450317746s) [upload=452901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:39:35.873139933 (516h10m35.744222978s) [upload=452901]: Done uploading segments... Dec 27 03:39:35.873157767 (516h10m35.744240802s) [upload=452901]: Upload finished. err: <nil> Duration: 55.566807732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.X] duration: 30.004126478s trace: https://jaeger.dev.storj.io/trace/293f1192e1dbaaa7 FAIL (2025-12-27 03:43:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 03:43:07.326984931 (516h14m7.198067966s) [upload=452955]: Starting upload Dec 27 03:43:07.327091716 (516h14m7.198174751s) [upload=452955]: Done starting upload Dec 27 03:43:07.327230780 (516h14m7.198313816s) [upload=452955]: Uploading segments... Dec 27 03:43:07.327362961 (516h14m7.198445996s) [upload=452955]: Got next segment. Inline: false Dec 27 03:43:07.327405603 (516h14m7.198488648s) [upload=452955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:43:07.327412596 (516h14m7.198495632s) [upload=452955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:43:07.570166186 (516h14m7.441249271s) [seg_pos={0 0}, upload=452955]: Begin upload segment... Dec 27 03:43:07.570347741 (516h14m7.441430796s) [seg_pos={0 0}, upload=452955]: Done begin upload segment. Dec 27 03:43:07.570376757 (516h14m7.441459792s) [noise=true , address=145.79.227.34:4905 , piece_id=2T7ZKAPHVVAL3WL2OVMMC4LPBXQPO7CRV2466YQ5IVKZMLSUPVBQ, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.570391805 (516h14m7.44147485s) [noise=true , address=145.79.227.11:4729 , piece_id=3M7ICUZ6HL43ZIGHJDZGDISVVGCMNDQXE37MBGFWEP2C77MM76AA, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.571082951 (516h14m7.442165996s) [noise=true , address=77.90.31.11:4554 , piece_id=XFZRE6OCYUF6O25ZJQDYWPZF5PNKJRQUT2MXWX5O2ZATJHKRO64A, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.571618792 (516h14m7.442701847s) [noise=true , address=145.79.227.16:4912 , piece_id=NFVC2XST4YIDJ6JP7PDUOIKIFA2QGHDOOIZUYN7IRRNHO3YF7LJQ, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.572146427 (516h14m7.443229492s) [noise=true , address=45.91.116.13:4153 , piece_id=UM24ZUU5H54V354YOEDMOIIXX35T4OLQUZDGKQTSUJWDZRXRYPQA, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.572708887 (516h14m7.443791922s) [noise=true , address=145.79.227.49:4339 , piece_id=F6NQY3MMWIGZFR7UZ5R5PSU4WWE6JX3EGKOU6ACUOC2NZ5GVSQSA, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.573256160 (516h14m7.444339255s) [noise=true , address=145.79.227.4:4185 , piece_id=F2MCB6NZ3I7E5Q6IEJEHAT2T53BVX5GFT7AYIWUE5USUMLR5ENBA, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.575391796 (516h14m7.446474851s) [noise=true , address=145.79.227.48:4303 , piece_id=G4KMIS37XSZ2YL65QG5KKHG4QJKDPZ4OZEMCC3XYEKUK3YC6KTTQ, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.579139022 (516h14m7.450222087s) [noise=true , address=45.91.116.12:4227 , piece_id=PZFSIAJRQR3G6WIAAALF3EZNDBB4STREXGP4OL67LF33GPF6XFQA, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.581767107 (516h14m7.452850142s) [noise=true , address=145.79.227.41:4085 , piece_id=2DTCLCIK3JZJVMFATOHKMZFEAWWBYPSEX3C6NRV4URF2JFLMI4LA, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.584522984 (516h14m7.455606019s) [noise=true , address=45.91.116.10:4083 , piece_id=6ESW242KEEFCEML4LSBTQEOGZHKLF2RIOYRDFYZSG5QSTNFQJTBA, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.586806051 (516h14m7.457889096s) [noise=true , address=45.91.116.8:4047 , piece_id=RLCBPWE7Q5TCVN7LD4RLMIW45CSCT6HFRNX52U4724SVPRDCCMOA, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.588064268 (516h14m7.459147313s) [noise=true , address=145.79.227.52:4446 , piece_id=DTYUC7ZKOVEZKNKAGGJ5JG7IW2L4BCXPJGLPTHWGIB32UWWBZ5XQ, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.590693654 (516h14m7.461776699s) [noise=true , address=77.90.31.14:4663 , piece_id=NLTMWKPHRVCNC6PCRBJ4K2G3EXTIPQKYN3L6CGHTFDUYYIWRQ3MA, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.593574059 (516h14m7.464657114s) [noise=true , address=145.79.227.47:4265 , piece_id=XYE54GWHILTUKZKZVJB56HLXX5G7GMIGRWLBWL4MWR72XIVUWQEQ, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:07.593590560 (516h14m7.464673605s) [noise=true , address=77.90.31.21:4880 , piece_id=JV4U3SUTIIAQGRGJVXTIQCWVERLSHGH7YD3PBQQZ2CMQXMHIYVVQ, seg_pos={0 0}, upload=452955]: Uploading piece... Dec 27 03:43:09.292927684 (516h14m9.164010739s) [upload=452955]: Next returned nil segment Dec 27 03:43:09.292948052 (516h14m9.164031097s) [upload=452955]: Waiting for error group managing segments... Dec 27 03:43:09.579286255 (516h14m9.4503693s) [noise=true , address=145.79.227.41:4085 , piece_id=2DTCLCIK3JZJVMFATOHKMZFEAWWBYPSEX3C6NRV4URF2JFLMI4LA, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:09.579933958 (516h14m9.451017003s) [noise=true , address=145.79.227.48:4303 , piece_id=G4KMIS37XSZ2YL65QG5KKHG4QJKDPZ4OZEMCC3XYEKUK3YC6KTTQ, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:09.588119145 (516h14m9.45920219s) [noise=true , address=145.79.227.52:4446 , piece_id=DTYUC7ZKOVEZKNKAGGJ5JG7IW2L4BCXPJGLPTHWGIB32UWWBZ5XQ, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:09.600342831 (516h14m9.471425866s) [noise=true , address=145.79.227.49:4339 , piece_id=F6NQY3MMWIGZFR7UZ5R5PSU4WWE6JX3EGKOU6ACUOC2NZ5GVSQSA, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:09.608268172 (516h14m9.479351217s) [noise=true , address=145.79.227.16:4912 , piece_id=NFVC2XST4YIDJ6JP7PDUOIKIFA2QGHDOOIZUYN7IRRNHO3YF7LJQ, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:09.610533025 (516h14m9.48161607s) [noise=true , address=145.79.227.11:4729 , piece_id=3M7ICUZ6HL43ZIGHJDZGDISVVGCMNDQXE37MBGFWEP2C77MM76AA, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:09.617374854 (516h14m9.488457899s) [noise=true , address=145.79.227.47:4265 , piece_id=XYE54GWHILTUKZKZVJB56HLXX5G7GMIGRWLBWL4MWR72XIVUWQEQ, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:09.847234791 (516h14m9.718317836s) [noise=true , address=45.91.116.8:4047 , piece_id=RLCBPWE7Q5TCVN7LD4RLMIW45CSCT6HFRNX52U4724SVPRDCCMOA, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:09.856702148 (516h14m9.727785183s) [noise=true , address=45.91.116.13:4153 , piece_id=UM24ZUU5H54V354YOEDMOIIXX35T4OLQUZDGKQTSUJWDZRXRYPQA, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:09.977371696 (516h14m9.848454741s) [noise=true , address=45.91.116.10:4083 , piece_id=6ESW242KEEFCEML4LSBTQEOGZHKLF2RIOYRDFYZSG5QSTNFQJTBA, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:10.074926669 (516h14m9.946009714s) [noise=true , address=77.90.31.14:4663 , piece_id=NLTMWKPHRVCNC6PCRBJ4K2G3EXTIPQKYN3L6CGHTFDUYYIWRQ3MA, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:10.141070259 (516h14m10.012153294s) [noise=true , address=77.90.31.11:4554 , piece_id=XFZRE6OCYUF6O25ZJQDYWPZF5PNKJRQUT2MXWX5O2ZATJHKRO64A, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:10.229673828 (516h14m10.100756863s) [noise=true , address=77.90.31.21:4880 , piece_id=JV4U3SUTIIAQGRGJVXTIQCWVERLSHGH7YD3PBQQZ2CMQXMHIYVVQ, seg_pos={0 0}, upload=452955]: Done uploading piece. err: <nil> Dec 27 03:43:10.229694297 (516h14m10.100777332s) [seg_pos={0 0}, upload=452955]: Segment reached optimal threshold of 13 pieces. Dec 27 03:43:10.229911271 (516h14m10.100994306s) [noise=true , address=145.79.227.4:4185 , piece_id=F2MCB6NZ3I7E5Q6IEJEHAT2T53BVX5GFT7AYIWUE5USUMLR5ENBA, seg_pos={0 0}, upload=452955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:33954->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:33954->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33954->145.79.227.4:4185: use of closed network connection Dec 27 03:43:10.229918073 (516h14m10.101001108s) [noise=true , address=145.79.227.4:4185 , piece_id=F2MCB6NZ3I7E5Q6IEJEHAT2T53BVX5GFT7AYIWUE5USUMLR5ENBA, seg_pos={0 0}, upload=452955]: Long tail cancellation due to optimal threshold Dec 27 03:43:10.230068269 (516h14m10.101151304s) [noise=true , address=145.79.227.34:4905 , piece_id=2T7ZKAPHVVAL3WL2OVMMC4LPBXQPO7CRV2466YQ5IVKZMLSUPVBQ, seg_pos={0 0}, upload=452955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 27 03:43:10.230092405 (516h14m10.10117545s) [noise=true , address=145.79.227.34:4905 , piece_id=2T7ZKAPHVVAL3WL2OVMMC4LPBXQPO7CRV2466YQ5IVKZMLSUPVBQ, seg_pos={0 0}, upload=452955]: Long tail cancellation due to optimal threshold Dec 27 03:43:10.230215639 (516h14m10.101298674s) [noise=true , address=45.91.116.12:4227 , piece_id=PZFSIAJRQR3G6WIAAALF3EZNDBB4STREXGP4OL67LF33GPF6XFQA, seg_pos={0 0}, upload=452955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 27 03:43:10.230222623 (516h14m10.101305658s) [noise=true , address=45.91.116.12:4227 , piece_id=PZFSIAJRQR3G6WIAAALF3EZNDBB4STREXGP4OL67LF33GPF6XFQA, seg_pos={0 0}, upload=452955]: Long tail cancellation due to optimal threshold Dec 27 03:43:10.230239064 (516h14m10.101322099s) [seg_pos={0 0}, upload=452955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:43:10.230323304 (516h14m10.101406349s) [upload=452955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:43:10.230327864 (516h14m10.101410899s) [upload=452955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:43:10.521864635 (516h14m10.39294767s) [upload=452955]: Done uploading segments... Dec 27 03:43:10.521884332 (516h14m10.392967367s) [upload=452955]: Upload finished. err: <nil> Duration: 30.004155774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 25.453421376s trace: https://jaeger.dev.storj.io/trace/2f1833ad3fc26997 FAIL (2025-12-27 03:43:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.046968512 (516h10m16.918051557s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047520183 (516h10m16.918603228s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047984678 (516h10m16.919067723s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048465613 (516h10m16.919548658s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048734075 (516h10m16.91981712s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049142162 (516h10m16.920225207s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049245980 (516h10m16.920329025s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049769577 (516h10m16.920852612s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050319855 (516h10m16.9214029s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050902745 (516h10m16.92198579s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051099419 (516h10m16.922182464s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051941683 (516h10m16.923024738s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.052719805 (516h10m16.92380284s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.054577933 (516h10m16.925660969s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.058395733 (516h10m16.929478778s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:18.699503155 (516h10m18.5705862s) [upload=452901]: Next returned nil segment Dec 27 03:39:18.699553180 (516h10m18.570636226s) [upload=452901]: Waiting for error group managing segments... Dec 27 03:39:19.134636692 (516h10m19.005719727s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.202132478 (516h10m19.073215523s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.203422765 (516h10m19.07450581s) [noise=true , address=45.91.116.14:4194 , piece_id=Z2K5GGTN367GZ6NLOKWTBTVD2HNYEFEAA2MDGSZ2YMZG3XAFGEZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.235358065 (516h10m19.106441101s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.243623684 (516h10m19.114706729s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.298044432 (516h10m19.169127477s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.523897079 (516h10m19.394980124s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:20.015944923 (516h10m19.887027968s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:21.684763342 (516h10m21.555846377s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:22.446094653 (516h10m22.317177698s) [noise=true , address=77.90.31.18:4813 , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:24.365452157 (516h10m24.236535202s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:31.513341529 (516h10m31.384424564s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:32.290856236 (516h10m32.161939301s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578538275 (516h10m35.44962131s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578557731 (516h10m35.449640766s) [seg_pos={0 0}, upload=452901]: Segment reached optimal threshold of 14 pieces. Dec 27 03:39:35.578728707 (516h10m35.449811742s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Dec 27 03:39:35.578734489 (516h10m35.449817524s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.578899161 (516h10m35.449982207s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo): protocol: rpc: dial tcp 145.79.227.29:4949: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.578904221 (516h10m35.449987266s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579127447 (516h10m35.450210482s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: read tcp 188.166.225.170:58082->145.79.227.25:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579135101 (516h10m35.450218136s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579158115 (516h10m35.45024115s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: read tcp 188.166.225.170:40188->145.79.227.40:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579162754 (516h10m35.450245799s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579173815 (516h10m35.45025686s) [seg_pos={0 0}, upload=452901]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:39:35.579230252 (516h10m35.450313287s) [upload=452901]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:39:35.579234711 (516h10m35.450317746s) [upload=452901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:39:35.873139933 (516h10m35.744222978s) [upload=452901]: Done uploading segments... Dec 27 03:39:35.873157767 (516h10m35.744240802s) [upload=452901]: Upload finished. err: <nil> Duration: 25.453434712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 46.032085871s trace: https://jaeger.dev.storj.io/trace/2f1833ad3fc26997 FAIL (2025-12-27 03:42:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.046968512 (516h10m16.918051557s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047520183 (516h10m16.918603228s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047984678 (516h10m16.919067723s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048465613 (516h10m16.919548658s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048734075 (516h10m16.91981712s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049142162 (516h10m16.920225207s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049245980 (516h10m16.920329025s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049769577 (516h10m16.920852612s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050319855 (516h10m16.9214029s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050902745 (516h10m16.92198579s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051099419 (516h10m16.922182464s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051941683 (516h10m16.923024738s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.052719805 (516h10m16.92380284s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.054577933 (516h10m16.925660969s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.058395733 (516h10m16.929478778s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:18.699503155 (516h10m18.5705862s) [upload=452901]: Next returned nil segment Dec 27 03:39:18.699553180 (516h10m18.570636226s) [upload=452901]: Waiting for error group managing segments... Dec 27 03:39:19.134636692 (516h10m19.005719727s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.202132478 (516h10m19.073215523s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.203422765 (516h10m19.07450581s) [noise=true , address=45.91.116.14:4194 , piece_id=Z2K5GGTN367GZ6NLOKWTBTVD2HNYEFEAA2MDGSZ2YMZG3XAFGEZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.235358065 (516h10m19.106441101s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.243623684 (516h10m19.114706729s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.298044432 (516h10m19.169127477s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.523897079 (516h10m19.394980124s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:20.015944923 (516h10m19.887027968s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:21.684763342 (516h10m21.555846377s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:22.446094653 (516h10m22.317177698s) [noise=true , address=77.90.31.18:4813 , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:24.365452157 (516h10m24.236535202s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:31.513341529 (516h10m31.384424564s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:32.290856236 (516h10m32.161939301s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578538275 (516h10m35.44962131s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578557731 (516h10m35.449640766s) [seg_pos={0 0}, upload=452901]: Segment reached optimal threshold of 14 pieces. Dec 27 03:39:35.578728707 (516h10m35.449811742s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Dec 27 03:39:35.578734489 (516h10m35.449817524s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.578899161 (516h10m35.449982207s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo): protocol: rpc: dial tcp 145.79.227.29:4949: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.578904221 (516h10m35.449987266s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579127447 (516h10m35.450210482s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: read tcp 188.166.225.170:58082->145.79.227.25:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579135101 (516h10m35.450218136s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579158115 (516h10m35.45024115s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: read tcp 188.166.225.170:40188->145.79.227.40:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579162754 (516h10m35.450245799s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579173815 (516h10m35.45025686s) [seg_pos={0 0}, upload=452901]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:39:35.579230252 (516h10m35.450313287s) [upload=452901]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:39:35.579234711 (516h10m35.450317746s) [upload=452901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:39:35.873139933 (516h10m35.744222978s) [upload=452901]: Done uploading segments... Dec 27 03:39:35.873157767 (516h10m35.744240802s) [upload=452901]: Upload finished. err: <nil> Duration: 46.032112151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX.] duration: 6.132936989s trace: https://jaeger.dev.storj.io/trace/76d6da37a67bf0ae FAIL (2025-12-27 03:42:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 27 03:41:24.894854236 (516h12m24.765937271s) [upload=452930]: Got next segment. Inline: false Dec 27 03:41:24.894909722 (516h12m24.765992757s) [upload=452930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:41:24.894916114 (516h12m24.765999159s) [upload=452930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:41:25.137770305 (516h12m25.00885334s) [seg_pos={0 0}, upload=452930]: Begin upload segment... Dec 27 03:41:25.137843284 (516h12m25.008926329s) [seg_pos={0 0}, upload=452930]: Done begin upload segment. Dec 27 03:41:25.137894390 (516h12m25.008977426s) [noise=true , address=145.79.227.39:4015 , piece_id=ZH77EBDEBUDG5WW7UP5FJLTQPDZQBPXBMCBU7IIRHSB5D7JSE6TQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.137944096 (516h12m25.009027331s) [noise=true , address=45.91.116.8:4046 , piece_id=JQQHO5H43BAUICWFNIQYJAD6RLK65YRD72OSEUM2H3XXO3MWFMPQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.138365688 (516h12m25.009448723s) [noise=true , address=145.79.227.5:4222 , piece_id=FWJIK6H3ZJ5PUQUJTYG6KL7KWZWKZ44TRS7CELAMT23I5O5BBD2A, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.138739941 (516h12m25.009822976s) [noise=true , address=145.79.227.38:4979 , piece_id=PLTWK3QWA3SAMDQBTB77H5GEW7E2NBYKAVDPREIKCBA565JTDEQA, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.139024603 (516h12m25.010107638s) [noise=true , address=145.79.227.49:4339 , piece_id=F7NX76IBJVK5WJWN3EZWVVTYCWYPFFALHJVENGR3NIWYIZD4H22A, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.139339413 (516h12m25.010422448s) [noise=true , address=145.79.227.51:4411 , piece_id=LDUS2IIBDJ63XDDC2FYAN4VOHFGQPU6C5LOXJVY7YGLDYXBIBS2Q, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.139652970 (516h12m25.010736005s) [noise=true , address=77.90.31.12:4590 , piece_id=TBIV3WK4CESRDDWOSQBHRYE3KNBI4TEKV23UCLNKG5UHX3CC3L2Q, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.139982116 (516h12m25.011065151s) [noise=true , address=145.79.227.11:4732 , piece_id=H4T74HA2JIJFPNNADE5VDWVUTZ5UWUAY7FP3QXXZFVM764EK2NLA, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.140373010 (516h12m25.011456045s) [noise=true , address=77.90.31.13:4627 , piece_id=YQVNVXVXIMKAN25RZSE76JZHDK7X4DIHB2W2RV25STPLTFJDLQMA, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.140701085 (516h12m25.01178413s) [noise=true , address=45.91.116.7:4010 , piece_id=RJIXW3F3W6ZSW4IUO4GXNKDX6UHTMSZPVFN3FRGNBRHSCJMAWAMQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.141736607 (516h12m25.012819642s) [noise=true , address=45.91.116.12:4225 , piece_id=C6MPE55HXTBJERFWPJHKJ2B3DRN2EH7QOYKJT4WO336PJNUYWMDQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.141762466 (516h12m25.012845501s) [noise=true , address=45.91.116.9:4262 , piece_id=DBKLJYKGRSEO57WQZWXHSB6MFF6D65QHPDO2O7TY7NDQPFG7JZNA, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.142869063 (516h12m25.013952098s) [noise=true , address=145.79.227.2:4115 , piece_id=YJ3DDQMAMN2RECRVTNFVXKE577V2LWRR55FOAR74AXETDDBSRQ2Q, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.143981071 (516h12m25.015064106s) [noise=true , address=145.79.227.3:4151 , piece_id=Y6XVJNNNM5ALVJS7XYLXIU4RVRB7RO4NBULSY3BVOPSSH3KW2DVQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.145122264 (516h12m25.016205299s) [noise=true , address=145.79.227.52:4447 , piece_id=PZMFCCVJWSEEREOBMQRTKMO5FSBAVN2KTCZXVG2HY7QORHLSZKOQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.146238990 (516h12m25.017322035s) [noise=true , address=45.91.116.10:4084 , piece_id=M6UOOJFJKVTVHZDQTCLA4LRFDBEHAZHUN7C2T4SH6EDDA4TZGWHQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:26.870952797 (516h12m26.742035832s) [upload=452930]: Next returned nil segment Dec 27 03:41:26.870970541 (516h12m26.742053576s) [upload=452930]: Waiting for error group managing segments... Dec 27 03:41:27.148619668 (516h12m27.019702713s) [noise=true , address=145.79.227.11:4732 , piece_id=H4T74HA2JIJFPNNADE5VDWVUTZ5UWUAY7FP3QXXZFVM764EK2NLA, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.153041939 (516h12m27.024124984s) [noise=true , address=145.79.227.39:4015 , piece_id=ZH77EBDEBUDG5WW7UP5FJLTQPDZQBPXBMCBU7IIRHSB5D7JSE6TQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.162952190 (516h12m27.034035225s) [noise=true , address=145.79.227.49:4339 , piece_id=F7NX76IBJVK5WJWN3EZWVVTYCWYPFFALHJVENGR3NIWYIZD4H22A, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.167865445 (516h12m27.03894848s) [noise=true , address=145.79.227.2:4115 , piece_id=YJ3DDQMAMN2RECRVTNFVXKE577V2LWRR55FOAR74AXETDDBSRQ2Q, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.169708274 (516h12m27.040791319s) [noise=true , address=145.79.227.3:4151 , piece_id=Y6XVJNNNM5ALVJS7XYLXIU4RVRB7RO4NBULSY3BVOPSSH3KW2DVQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.171669489 (516h12m27.042752525s) [noise=true , address=145.79.227.38:4979 , piece_id=PLTWK3QWA3SAMDQBTB77H5GEW7E2NBYKAVDPREIKCBA565JTDEQA, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.171884829 (516h12m27.042967864s) [noise=true , address=145.79.227.52:4447 , piece_id=PZMFCCVJWSEEREOBMQRTKMO5FSBAVN2KTCZXVG2HY7QORHLSZKOQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.174765174 (516h12m27.045848219s) [noise=true , address=145.79.227.51:4411 , piece_id=LDUS2IIBDJ63XDDC2FYAN4VOHFGQPU6C5LOXJVY7YGLDYXBIBS2Q, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.468055806 (516h12m27.339138851s) [noise=true , address=45.91.116.9:4262 , piece_id=DBKLJYKGRSEO57WQZWXHSB6MFF6D65QHPDO2O7TY7NDQPFG7JZNA, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.502887552 (516h12m27.373970587s) [noise=true , address=45.91.116.10:4084 , piece_id=M6UOOJFJKVTVHZDQTCLA4LRFDBEHAZHUN7C2T4SH6EDDA4TZGWHQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.555945985 (516h12m27.42702902s) [noise=true , address=77.90.31.12:4590 , piece_id=TBIV3WK4CESRDDWOSQBHRYE3KNBI4TEKV23UCLNKG5UHX3CC3L2Q, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.569993363 (516h12m27.441076408s) [noise=true , address=45.91.116.7:4010 , piece_id=RJIXW3F3W6ZSW4IUO4GXNKDX6UHTMSZPVFN3FRGNBRHSCJMAWAMQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.674858277 (516h12m27.545941312s) [noise=true , address=45.91.116.8:4046 , piece_id=JQQHO5H43BAUICWFNIQYJAD6RLK65YRD72OSEUM2H3XXO3MWFMPQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.674875279 (516h12m27.545958314s) [seg_pos={0 0}, upload=452930]: Segment reached optimal threshold of 13 pieces. Dec 27 03:41:27.675035264 (516h12m27.546118299s) [noise=true , address=145.79.227.5:4222 , piece_id=FWJIK6H3ZJ5PUQUJTYG6KL7KWZWKZ44TRS7CELAMT23I5O5BBD2A, seg_pos={0 0}, upload=452930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:52306->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:52306->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52306->145.79.227.5:4222: use of closed network connection Dec 27 03:41:27.675040934 (516h12m27.546123969s) [noise=true , address=145.79.227.5:4222 , piece_id=FWJIK6H3ZJ5PUQUJTYG6KL7KWZWKZ44TRS7CELAMT23I5O5BBD2A, seg_pos={0 0}, upload=452930]: Long tail cancellation due to optimal threshold Dec 27 03:41:27.675204646 (516h12m27.546287681s) [noise=true , address=77.90.31.13:4627 , piece_id=YQVNVXVXIMKAN25RZSE76JZHDK7X4DIHB2W2RV25STPLTFJDLQMA, seg_pos={0 0}, upload=452930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50802->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50802->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50802->77.90.31.13:4627: use of closed network connection Dec 27 03:41:27.675210327 (516h12m27.546293362s) [noise=true , address=77.90.31.13:4627 , piece_id=YQVNVXVXIMKAN25RZSE76JZHDK7X4DIHB2W2RV25STPLTFJDLQMA, seg_pos={0 0}, upload=452930]: Long tail cancellation due to optimal threshold Dec 27 03:41:27.675293044 (516h12m27.546376089s) [noise=true , address=45.91.116.12:4225 , piece_id=C6MPE55HXTBJERFWPJHKJ2B3DRN2EH7QOYKJT4WO336PJNUYWMDQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 27 03:41:27.675312822 (516h12m27.546395877s) [noise=true , address=45.91.116.12:4225 , piece_id=C6MPE55HXTBJERFWPJHKJ2B3DRN2EH7QOYKJT4WO336PJNUYWMDQ, seg_pos={0 0}, upload=452930]: Long tail cancellation due to optimal threshold Dec 27 03:41:27.675344022 (516h12m27.546427057s) [seg_pos={0 0}, upload=452930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:41:27.675432880 (516h12m27.546515916s) [upload=452930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:41:27.675439484 (516h12m27.546522529s) [upload=452930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:41:27.971680523 (516h12m27.842763568s) [upload=452930]: Done uploading segments... Dec 27 03:41:27.971702174 (516h12m27.84278522s) [upload=452930]: Upload finished. err: <nil> Duration: 6.13296448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 38.577988926s trace: https://jaeger.dev.storj.io/trace/2f1833ad3fc26997 FAIL (2025-12-27 03:42:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.046968512 (516h10m16.918051557s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047520183 (516h10m16.918603228s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047984678 (516h10m16.919067723s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048465613 (516h10m16.919548658s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048734075 (516h10m16.91981712s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049142162 (516h10m16.920225207s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049245980 (516h10m16.920329025s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049769577 (516h10m16.920852612s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050319855 (516h10m16.9214029s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050902745 (516h10m16.92198579s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051099419 (516h10m16.922182464s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051941683 (516h10m16.923024738s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.052719805 (516h10m16.92380284s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.054577933 (516h10m16.925660969s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.058395733 (516h10m16.929478778s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:18.699503155 (516h10m18.5705862s) [upload=452901]: Next returned nil segment Dec 27 03:39:18.699553180 (516h10m18.570636226s) [upload=452901]: Waiting for error group managing segments... Dec 27 03:39:19.134636692 (516h10m19.005719727s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.202132478 (516h10m19.073215523s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.203422765 (516h10m19.07450581s) [noise=true , address=45.91.116.14:4194 , piece_id=Z2K5GGTN367GZ6NLOKWTBTVD2HNYEFEAA2MDGSZ2YMZG3XAFGEZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.235358065 (516h10m19.106441101s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.243623684 (516h10m19.114706729s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.298044432 (516h10m19.169127477s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.523897079 (516h10m19.394980124s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:20.015944923 (516h10m19.887027968s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:21.684763342 (516h10m21.555846377s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:22.446094653 (516h10m22.317177698s) [noise=true , address=77.90.31.18:4813 , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:24.365452157 (516h10m24.236535202s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:31.513341529 (516h10m31.384424564s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:32.290856236 (516h10m32.161939301s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578538275 (516h10m35.44962131s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578557731 (516h10m35.449640766s) [seg_pos={0 0}, upload=452901]: Segment reached optimal threshold of 14 pieces. Dec 27 03:39:35.578728707 (516h10m35.449811742s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Dec 27 03:39:35.578734489 (516h10m35.449817524s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.578899161 (516h10m35.449982207s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo): protocol: rpc: dial tcp 145.79.227.29:4949: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.578904221 (516h10m35.449987266s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579127447 (516h10m35.450210482s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: read tcp 188.166.225.170:58082->145.79.227.25:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579135101 (516h10m35.450218136s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579158115 (516h10m35.45024115s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: read tcp 188.166.225.170:40188->145.79.227.40:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579162754 (516h10m35.450245799s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579173815 (516h10m35.45025686s) [seg_pos={0 0}, upload=452901]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:39:35.579230252 (516h10m35.450313287s) [upload=452901]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:39:35.579234711 (516h10m35.450317746s) [upload=452901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:39:35.873139933 (516h10m35.744222978s) [upload=452901]: Done uploading segments... Dec 27 03:39:35.873157767 (516h10m35.744240802s) [upload=452901]: Upload finished. err: <nil> Duration: 38.57803191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.XX] duration: 30.003050054s trace: https://jaeger.dev.storj.io/trace/76d6da37a67bf0ae FAIL (2025-12-27 03:41:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 27 03:41:24.894854236 (516h12m24.765937271s) [upload=452930]: Got next segment. Inline: false Dec 27 03:41:24.894909722 (516h12m24.765992757s) [upload=452930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:41:24.894916114 (516h12m24.765999159s) [upload=452930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:41:25.137770305 (516h12m25.00885334s) [seg_pos={0 0}, upload=452930]: Begin upload segment... Dec 27 03:41:25.137843284 (516h12m25.008926329s) [seg_pos={0 0}, upload=452930]: Done begin upload segment. Dec 27 03:41:25.137894390 (516h12m25.008977426s) [noise=true , address=145.79.227.39:4015 , piece_id=ZH77EBDEBUDG5WW7UP5FJLTQPDZQBPXBMCBU7IIRHSB5D7JSE6TQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.137944096 (516h12m25.009027331s) [noise=true , address=45.91.116.8:4046 , piece_id=JQQHO5H43BAUICWFNIQYJAD6RLK65YRD72OSEUM2H3XXO3MWFMPQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.138365688 (516h12m25.009448723s) [noise=true , address=145.79.227.5:4222 , piece_id=FWJIK6H3ZJ5PUQUJTYG6KL7KWZWKZ44TRS7CELAMT23I5O5BBD2A, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.138739941 (516h12m25.009822976s) [noise=true , address=145.79.227.38:4979 , piece_id=PLTWK3QWA3SAMDQBTB77H5GEW7E2NBYKAVDPREIKCBA565JTDEQA, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.139024603 (516h12m25.010107638s) [noise=true , address=145.79.227.49:4339 , piece_id=F7NX76IBJVK5WJWN3EZWVVTYCWYPFFALHJVENGR3NIWYIZD4H22A, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.139339413 (516h12m25.010422448s) [noise=true , address=145.79.227.51:4411 , piece_id=LDUS2IIBDJ63XDDC2FYAN4VOHFGQPU6C5LOXJVY7YGLDYXBIBS2Q, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.139652970 (516h12m25.010736005s) [noise=true , address=77.90.31.12:4590 , piece_id=TBIV3WK4CESRDDWOSQBHRYE3KNBI4TEKV23UCLNKG5UHX3CC3L2Q, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.139982116 (516h12m25.011065151s) [noise=true , address=145.79.227.11:4732 , piece_id=H4T74HA2JIJFPNNADE5VDWVUTZ5UWUAY7FP3QXXZFVM764EK2NLA, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.140373010 (516h12m25.011456045s) [noise=true , address=77.90.31.13:4627 , piece_id=YQVNVXVXIMKAN25RZSE76JZHDK7X4DIHB2W2RV25STPLTFJDLQMA, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.140701085 (516h12m25.01178413s) [noise=true , address=45.91.116.7:4010 , piece_id=RJIXW3F3W6ZSW4IUO4GXNKDX6UHTMSZPVFN3FRGNBRHSCJMAWAMQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.141736607 (516h12m25.012819642s) [noise=true , address=45.91.116.12:4225 , piece_id=C6MPE55HXTBJERFWPJHKJ2B3DRN2EH7QOYKJT4WO336PJNUYWMDQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.141762466 (516h12m25.012845501s) [noise=true , address=45.91.116.9:4262 , piece_id=DBKLJYKGRSEO57WQZWXHSB6MFF6D65QHPDO2O7TY7NDQPFG7JZNA, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.142869063 (516h12m25.013952098s) [noise=true , address=145.79.227.2:4115 , piece_id=YJ3DDQMAMN2RECRVTNFVXKE577V2LWRR55FOAR74AXETDDBSRQ2Q, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.143981071 (516h12m25.015064106s) [noise=true , address=145.79.227.3:4151 , piece_id=Y6XVJNNNM5ALVJS7XYLXIU4RVRB7RO4NBULSY3BVOPSSH3KW2DVQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.145122264 (516h12m25.016205299s) [noise=true , address=145.79.227.52:4447 , piece_id=PZMFCCVJWSEEREOBMQRTKMO5FSBAVN2KTCZXVG2HY7QORHLSZKOQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:25.146238990 (516h12m25.017322035s) [noise=true , address=45.91.116.10:4084 , piece_id=M6UOOJFJKVTVHZDQTCLA4LRFDBEHAZHUN7C2T4SH6EDDA4TZGWHQ, seg_pos={0 0}, upload=452930]: Uploading piece... Dec 27 03:41:26.870952797 (516h12m26.742035832s) [upload=452930]: Next returned nil segment Dec 27 03:41:26.870970541 (516h12m26.742053576s) [upload=452930]: Waiting for error group managing segments... Dec 27 03:41:27.148619668 (516h12m27.019702713s) [noise=true , address=145.79.227.11:4732 , piece_id=H4T74HA2JIJFPNNADE5VDWVUTZ5UWUAY7FP3QXXZFVM764EK2NLA, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.153041939 (516h12m27.024124984s) [noise=true , address=145.79.227.39:4015 , piece_id=ZH77EBDEBUDG5WW7UP5FJLTQPDZQBPXBMCBU7IIRHSB5D7JSE6TQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.162952190 (516h12m27.034035225s) [noise=true , address=145.79.227.49:4339 , piece_id=F7NX76IBJVK5WJWN3EZWVVTYCWYPFFALHJVENGR3NIWYIZD4H22A, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.167865445 (516h12m27.03894848s) [noise=true , address=145.79.227.2:4115 , piece_id=YJ3DDQMAMN2RECRVTNFVXKE577V2LWRR55FOAR74AXETDDBSRQ2Q, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.169708274 (516h12m27.040791319s) [noise=true , address=145.79.227.3:4151 , piece_id=Y6XVJNNNM5ALVJS7XYLXIU4RVRB7RO4NBULSY3BVOPSSH3KW2DVQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.171669489 (516h12m27.042752525s) [noise=true , address=145.79.227.38:4979 , piece_id=PLTWK3QWA3SAMDQBTB77H5GEW7E2NBYKAVDPREIKCBA565JTDEQA, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.171884829 (516h12m27.042967864s) [noise=true , address=145.79.227.52:4447 , piece_id=PZMFCCVJWSEEREOBMQRTKMO5FSBAVN2KTCZXVG2HY7QORHLSZKOQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.174765174 (516h12m27.045848219s) [noise=true , address=145.79.227.51:4411 , piece_id=LDUS2IIBDJ63XDDC2FYAN4VOHFGQPU6C5LOXJVY7YGLDYXBIBS2Q, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.468055806 (516h12m27.339138851s) [noise=true , address=45.91.116.9:4262 , piece_id=DBKLJYKGRSEO57WQZWXHSB6MFF6D65QHPDO2O7TY7NDQPFG7JZNA, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.502887552 (516h12m27.373970587s) [noise=true , address=45.91.116.10:4084 , piece_id=M6UOOJFJKVTVHZDQTCLA4LRFDBEHAZHUN7C2T4SH6EDDA4TZGWHQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.555945985 (516h12m27.42702902s) [noise=true , address=77.90.31.12:4590 , piece_id=TBIV3WK4CESRDDWOSQBHRYE3KNBI4TEKV23UCLNKG5UHX3CC3L2Q, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.569993363 (516h12m27.441076408s) [noise=true , address=45.91.116.7:4010 , piece_id=RJIXW3F3W6ZSW4IUO4GXNKDX6UHTMSZPVFN3FRGNBRHSCJMAWAMQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.674858277 (516h12m27.545941312s) [noise=true , address=45.91.116.8:4046 , piece_id=JQQHO5H43BAUICWFNIQYJAD6RLK65YRD72OSEUM2H3XXO3MWFMPQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: <nil> Dec 27 03:41:27.674875279 (516h12m27.545958314s) [seg_pos={0 0}, upload=452930]: Segment reached optimal threshold of 13 pieces. Dec 27 03:41:27.675035264 (516h12m27.546118299s) [noise=true , address=145.79.227.5:4222 , piece_id=FWJIK6H3ZJ5PUQUJTYG6KL7KWZWKZ44TRS7CELAMT23I5O5BBD2A, seg_pos={0 0}, upload=452930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:52306->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:52306->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52306->145.79.227.5:4222: use of closed network connection Dec 27 03:41:27.675040934 (516h12m27.546123969s) [noise=true , address=145.79.227.5:4222 , piece_id=FWJIK6H3ZJ5PUQUJTYG6KL7KWZWKZ44TRS7CELAMT23I5O5BBD2A, seg_pos={0 0}, upload=452930]: Long tail cancellation due to optimal threshold Dec 27 03:41:27.675204646 (516h12m27.546287681s) [noise=true , address=77.90.31.13:4627 , piece_id=YQVNVXVXIMKAN25RZSE76JZHDK7X4DIHB2W2RV25STPLTFJDLQMA, seg_pos={0 0}, upload=452930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50802->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50802->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50802->77.90.31.13:4627: use of closed network connection Dec 27 03:41:27.675210327 (516h12m27.546293362s) [noise=true , address=77.90.31.13:4627 , piece_id=YQVNVXVXIMKAN25RZSE76JZHDK7X4DIHB2W2RV25STPLTFJDLQMA, seg_pos={0 0}, upload=452930]: Long tail cancellation due to optimal threshold Dec 27 03:41:27.675293044 (516h12m27.546376089s) [noise=true , address=45.91.116.12:4225 , piece_id=C6MPE55HXTBJERFWPJHKJ2B3DRN2EH7QOYKJT4WO336PJNUYWMDQ, seg_pos={0 0}, upload=452930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 27 03:41:27.675312822 (516h12m27.546395877s) [noise=true , address=45.91.116.12:4225 , piece_id=C6MPE55HXTBJERFWPJHKJ2B3DRN2EH7QOYKJT4WO336PJNUYWMDQ, seg_pos={0 0}, upload=452930]: Long tail cancellation due to optimal threshold Dec 27 03:41:27.675344022 (516h12m27.546427057s) [seg_pos={0 0}, upload=452930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:41:27.675432880 (516h12m27.546515916s) [upload=452930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:41:27.675439484 (516h12m27.546522529s) [upload=452930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:41:27.971680523 (516h12m27.842763568s) [upload=452930]: Done uploading segments... Dec 27 03:41:27.971702174 (516h12m27.84278522s) [upload=452930]: Upload finished. err: <nil> Duration: 30.003073037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [X.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.X] duration: 30.004778883s trace: https://jaeger.dev.storj.io/trace/7e6912caa972b46e FAIL (2025-12-27 03:41:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DYGEZ6SU7QCW5H3VRRTM3LF7EXUYDMPXKYQD7IQHIYV7ND6YEB4Q) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: AKJCRPXE3CHYB7JUMPOC5RL6HH7MVVACZONNPHQNWRP4U2F4AIHQ) context deadline exceeded ======LOGS====== g segments... Dec 27 03:40:37.718721302 (516h11m37.589804347s) [upload=452922]: Got next segment. Inline: false Dec 27 03:40:37.718780004 (516h11m37.589863049s) [upload=452922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:40:37.718785765 (516h11m37.5898688s) [upload=452922]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:40:37.962004750 (516h11m37.833087795s) [seg_pos={0 0}, upload=452922]: Begin upload segment... Dec 27 03:40:37.962079552 (516h11m37.833162587s) [seg_pos={0 0}, upload=452922]: Done begin upload segment. Dec 27 03:40:37.962120129 (516h11m37.833203164s) [noise=true , address=45.91.116.11:4119 , piece_id=DYGEZ6SU7QCW5H3VRRTM3LF7EXUYDMPXKYQD7IQHIYV7ND6YEB4Q, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.962936003 (516h11m37.834019038s) [noise=true , address=145.79.227.42:4121 , piece_id=YHRCEALVG5YEOP5WLS6VBUJSFN3EL7HUZL2OXR5GWI35V55UHHWQ, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.964500382 (516h11m37.835583417s) [noise=true , address=45.91.116.13:4154 , piece_id=WYKEM4R3CJQZ56Q532XPP6AU3OYO4R6VWKRDFNBIQ45VG3BVS5LQ, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.964901355 (516h11m37.83598439s) [noise=true , address=45.91.116.5:4298 , piece_id=GGAJGE7WTSFSVHMET6YBXOQAFXWHDVORQNJ2OWOBJSKBCRDITNLQ, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.966455335 (516h11m37.83753838s) [noise=true , address=145.79.227.51:4410 , piece_id=ETHW7FEYIUJZ77PRJBOA534UIYOFE2ZW5D6NWVWM63OXWXYLORTA, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.968541317 (516h11m37.839624352s) [noise=true , address=145.79.227.11:4729 , piece_id=MUXERDJ4Z62ZEQXK3JTPTWU5LLDLWYXHLU6DMIQRLMK2W5K4NSEA, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.969378070 (516h11m37.840461105s) [noise=true , address=145.79.227.45:4229 , piece_id=3TTHUCEMXCPCPMR5CDYV7T3DGPON3E6J5I3HYMOJALBWSAQ65EUQ, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.969777571 (516h11m37.840860616s) [noise=true , address=77.90.31.13:4627 , piece_id=UG56LMOKAZJQBRGKGXOCA57YRF5X5KU74I4JT3X2KIYO53MZGDRA, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.970880011 (516h11m37.841963076s) [noise=true , address=145.79.227.36:4519 , piece_id=QDVJZUZRBCDT5I6YVLURAMTOSEO72VOJBFTN75MQ44FGGGUGNPDQ, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.971201813 (516h11m37.842284859s) [noise=true , address=77.90.31.10:4519 , piece_id=AKJCRPXE3CHYB7JUMPOC5RL6HH7MVVACZONNPHQNWRP4U2F4AIHQ, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.972484907 (516h11m37.843567942s) [noise=true , address=145.79.227.15:4873 , piece_id=SIOSUPSYJBDKMJCX3BUSVTH4P5UEQ5MDG7IY5PPWTOCPR3WZB27Q, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.973722713 (516h11m37.844805758s) [noise=true , address=45.91.116.6:4973 , piece_id=XCV5SHPJK6RWC3YGZSP5D4DAEXPHEPZII2HMKJRQRO5XOSZ6W4VA, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.974376889 (516h11m37.845459934s) [noise=true , address=145.79.227.2:4114 , piece_id=5EOGB2WQ35LXSFITD5ITMDJ2DGPYMWNNIOJLNGG7HFW7IUAYPCTA, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.975607061 (516h11m37.846690096s) [noise=true , address=145.79.227.34:4905 , piece_id=EBIMMU5FXKJ3CPTPU6FQMMSX5KL33R5XCZCIQRWWY3BWPC7QVVZA, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.976028565 (516h11m37.84711161s) [noise=true , address=145.79.227.37:4565 , piece_id=3MY64FEQ7IYRZJVYIFS57VN7ASU5U2YG5REKTUUNQB2AT66R47TA, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:37.976878944 (516h11m37.847961979s) [noise=true , address=145.79.227.22:4511 , piece_id=RJXZPGZVMXIWMF6PHPBEJEDK4C4J52757PSEE62TDBL4AKEYTQCQ, seg_pos={0 0}, upload=452922]: Uploading piece... Dec 27 03:40:39.675211674 (516h11m39.546294719s) [upload=452922]: Next returned nil segment Dec 27 03:40:39.675224519 (516h11m39.546307554s) [upload=452922]: Waiting for error group managing segments... Dec 27 03:40:39.959205793 (516h11m39.830288838s) [noise=true , address=145.79.227.51:4410 , piece_id=ETHW7FEYIUJZ77PRJBOA534UIYOFE2ZW5D6NWVWM63OXWXYLORTA, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:39.966605484 (516h11m39.837688519s) [noise=true , address=145.79.227.11:4729 , piece_id=MUXERDJ4Z62ZEQXK3JTPTWU5LLDLWYXHLU6DMIQRLMK2W5K4NSEA, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:39.979462456 (516h11m39.850545501s) [noise=true , address=145.79.227.45:4229 , piece_id=3TTHUCEMXCPCPMR5CDYV7T3DGPON3E6J5I3HYMOJALBWSAQ65EUQ, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:39.980661499 (516h11m39.851744534s) [noise=true , address=145.79.227.42:4121 , piece_id=YHRCEALVG5YEOP5WLS6VBUJSFN3EL7HUZL2OXR5GWI35V55UHHWQ, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:39.982737011 (516h11m39.853820046s) [noise=true , address=145.79.227.36:4519 , piece_id=QDVJZUZRBCDT5I6YVLURAMTOSEO72VOJBFTN75MQ44FGGGUGNPDQ, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:39.988952487 (516h11m39.860035532s) [noise=true , address=145.79.227.15:4873 , piece_id=SIOSUPSYJBDKMJCX3BUSVTH4P5UEQ5MDG7IY5PPWTOCPR3WZB27Q, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:40.006626621 (516h11m39.877709656s) [noise=true , address=145.79.227.37:4565 , piece_id=3MY64FEQ7IYRZJVYIFS57VN7ASU5U2YG5REKTUUNQB2AT66R47TA, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:40.016283861 (516h11m39.887366906s) [noise=true , address=145.79.227.22:4511 , piece_id=RJXZPGZVMXIWMF6PHPBEJEDK4C4J52757PSEE62TDBL4AKEYTQCQ, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:40.247524677 (516h11m40.118607712s) [noise=true , address=77.90.31.13:4627 , piece_id=UG56LMOKAZJQBRGKGXOCA57YRF5X5KU74I4JT3X2KIYO53MZGDRA, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:40.313996843 (516h11m40.185079888s) [noise=true , address=45.91.116.5:4298 , piece_id=GGAJGE7WTSFSVHMET6YBXOQAFXWHDVORQNJ2OWOBJSKBCRDITNLQ, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:40.483807385 (516h11m40.35489043s) [noise=true , address=145.79.227.34:4905 , piece_id=EBIMMU5FXKJ3CPTPU6FQMMSX5KL33R5XCZCIQRWWY3BWPC7QVVZA, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:40.525351964 (516h11m40.396434999s) [noise=true , address=45.91.116.11:4119 , piece_id=DYGEZ6SU7QCW5H3VRRTM3LF7EXUYDMPXKYQD7IQHIYV7ND6YEB4Q, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:40.571687575 (516h11m40.44277061s) [noise=true , address=77.90.31.10:4519 , piece_id=AKJCRPXE3CHYB7JUMPOC5RL6HH7MVVACZONNPHQNWRP4U2F4AIHQ, seg_pos={0 0}, upload=452922]: Done uploading piece. err: <nil> Dec 27 03:40:40.571710088 (516h11m40.442793133s) [seg_pos={0 0}, upload=452922]: Segment reached optimal threshold of 13 pieces. Dec 27 03:40:40.571882266 (516h11m40.442965311s) [noise=true , address=145.79.227.2:4114 , piece_id=5EOGB2WQ35LXSFITD5ITMDJ2DGPYMWNNIOJLNGG7HFW7IUAYPCTA, seg_pos={0 0}, upload=452922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:58860->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:58860->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58860->145.79.227.2:4114: use of closed network connection Dec 27 03:40:40.571887916 (516h11m40.442970951s) [noise=true , address=145.79.227.2:4114 , piece_id=5EOGB2WQ35LXSFITD5ITMDJ2DGPYMWNNIOJLNGG7HFW7IUAYPCTA, seg_pos={0 0}, upload=452922]: Long tail cancellation due to optimal threshold Dec 27 03:40:40.571986314 (516h11m40.443069359s) [noise=true , address=45.91.116.6:4973 , piece_id=XCV5SHPJK6RWC3YGZSP5D4DAEXPHEPZII2HMKJRQRO5XOSZ6W4VA, seg_pos={0 0}, upload=452922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:60816->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:60816->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60816->45.91.116.6:4973: use of closed network connection Dec 27 03:40:40.571995581 (516h11m40.443078616s) [noise=true , address=45.91.116.6:4973 , piece_id=XCV5SHPJK6RWC3YGZSP5D4DAEXPHEPZII2HMKJRQRO5XOSZ6W4VA, seg_pos={0 0}, upload=452922]: Long tail cancellation due to optimal threshold Dec 27 03:40:40.572128495 (516h11m40.44321154s) [noise=true , address=45.91.116.13:4154 , piece_id=WYKEM4R3CJQZ56Q532XPP6AU3OYO4R6VWKRDFNBIQ45VG3BVS5LQ, seg_pos={0 0}, upload=452922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 03:40:40.572135979 (516h11m40.443219024s) [noise=true , address=45.91.116.13:4154 , piece_id=WYKEM4R3CJQZ56Q532XPP6AU3OYO4R6VWKRDFNBIQ45VG3BVS5LQ, seg_pos={0 0}, upload=452922]: Long tail cancellation due to optimal threshold Dec 27 03:40:40.572158953 (516h11m40.443241998s) [seg_pos={0 0}, upload=452922]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:40:40.572252752 (516h11m40.443335787s) [upload=452922]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:40:40.572271066 (516h11m40.443354121s) [upload=452922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:40:40.858075733 (516h11m40.729158778s) [upload=452922]: Done uploading segments... Dec 27 03:40:40.858096723 (516h11m40.729179769s) [upload=452922]: Upload finished. err: <nil> Duration: 30.004805805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 15.151054958s trace: https://jaeger.dev.storj.io/trace/2f1833ad3fc26997 FAIL (2025-12-27 03:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.046968512 (516h10m16.918051557s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047520183 (516h10m16.918603228s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047984678 (516h10m16.919067723s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048465613 (516h10m16.919548658s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048734075 (516h10m16.91981712s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049142162 (516h10m16.920225207s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049245980 (516h10m16.920329025s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049769577 (516h10m16.920852612s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050319855 (516h10m16.9214029s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050902745 (516h10m16.92198579s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051099419 (516h10m16.922182464s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051941683 (516h10m16.923024738s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.052719805 (516h10m16.92380284s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.054577933 (516h10m16.925660969s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.058395733 (516h10m16.929478778s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:18.699503155 (516h10m18.5705862s) [upload=452901]: Next returned nil segment Dec 27 03:39:18.699553180 (516h10m18.570636226s) [upload=452901]: Waiting for error group managing segments... Dec 27 03:39:19.134636692 (516h10m19.005719727s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.202132478 (516h10m19.073215523s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.203422765 (516h10m19.07450581s) [noise=true , address=45.91.116.14:4194 , piece_id=Z2K5GGTN367GZ6NLOKWTBTVD2HNYEFEAA2MDGSZ2YMZG3XAFGEZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.235358065 (516h10m19.106441101s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.243623684 (516h10m19.114706729s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.298044432 (516h10m19.169127477s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.523897079 (516h10m19.394980124s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:20.015944923 (516h10m19.887027968s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:21.684763342 (516h10m21.555846377s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:22.446094653 (516h10m22.317177698s) [noise=true , address=77.90.31.18:4813 , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:24.365452157 (516h10m24.236535202s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:31.513341529 (516h10m31.384424564s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:32.290856236 (516h10m32.161939301s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578538275 (516h10m35.44962131s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578557731 (516h10m35.449640766s) [seg_pos={0 0}, upload=452901]: Segment reached optimal threshold of 14 pieces. Dec 27 03:39:35.578728707 (516h10m35.449811742s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Dec 27 03:39:35.578734489 (516h10m35.449817524s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.578899161 (516h10m35.449982207s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo): protocol: rpc: dial tcp 145.79.227.29:4949: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.578904221 (516h10m35.449987266s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579127447 (516h10m35.450210482s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: read tcp 188.166.225.170:58082->145.79.227.25:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579135101 (516h10m35.450218136s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579158115 (516h10m35.45024115s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: read tcp 188.166.225.170:40188->145.79.227.40:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579162754 (516h10m35.450245799s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579173815 (516h10m35.45025686s) [seg_pos={0 0}, upload=452901]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:39:35.579230252 (516h10m35.450313287s) [upload=452901]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:39:35.579234711 (516h10m35.450317746s) [upload=452901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:39:35.873139933 (516h10m35.744222978s) [upload=452901]: Done uploading segments... Dec 27 03:39:35.873157767 (516h10m35.744240802s) [upload=452901]: Upload finished. err: <nil> Duration: 15.151086588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003577986s trace: https://jaeger.dev.storj.io/trace/2f1833ad3fc26997 FAIL (2025-12-27 03:40:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.046968512 (516h10m16.918051557s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047520183 (516h10m16.918603228s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.047984678 (516h10m16.919067723s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048465613 (516h10m16.919548658s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.048734075 (516h10m16.91981712s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049142162 (516h10m16.920225207s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049245980 (516h10m16.920329025s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.049769577 (516h10m16.920852612s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050319855 (516h10m16.9214029s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.050902745 (516h10m16.92198579s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051099419 (516h10m16.922182464s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.051941683 (516h10m16.923024738s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.052719805 (516h10m16.92380284s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.054577933 (516h10m16.925660969s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:17.058395733 (516h10m16.929478778s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Uploading piece... Dec 27 03:39:18.699503155 (516h10m18.5705862s) [upload=452901]: Next returned nil segment Dec 27 03:39:18.699553180 (516h10m18.570636226s) [upload=452901]: Waiting for error group managing segments... Dec 27 03:39:19.134636692 (516h10m19.005719727s) [noise=true , address=5.231.73.2:4001 , piece_id=IQVFPYFPRBIWYQES2YAJJJNFDCYEOM3XSYVHMYBS24BWVFXYDO7Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.202132478 (516h10m19.073215523s) [noise=true , address=45.91.116.9:4291 , piece_id=SS546VHYLT4RZVBRY4ED6HIYBHPMJYMPMHCXAZSOUY2LUV3M5O6A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.203422765 (516h10m19.07450581s) [noise=true , address=45.91.116.14:4194 , piece_id=Z2K5GGTN367GZ6NLOKWTBTVD2HNYEFEAA2MDGSZ2YMZG3XAFGEZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.235358065 (516h10m19.106441101s) [noise=true , address=45.91.116.8:4059 , piece_id=LEAFJTXBT6KB65TM7F5GLLOGPYWW3QOOZVRIYDMMIUH6AR6LT4KQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.243623684 (516h10m19.114706729s) [noise=true , address=77.90.31.17:4779 , piece_id=BOOR7RBTDD27WA37XWOONDHAVVZ7KNYF7O3YIMAJS6LKLZ75HUSA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.298044432 (516h10m19.169127477s) [noise=true , address=77.90.31.13:4637 , piece_id=V5F5UDUF4XHTJQD7IEZ55MOY6XPECU2JEDFN6MTM7OTWSIXB3XAQ, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:19.523897079 (516h10m19.394980124s) [noise=true , address=45.91.116.4:4967 , piece_id=WYJAA65RKFQVKNH2575KUEPJSOG4Z3BHLIFUOOZS32WZ5QKCDOUA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:20.015944923 (516h10m19.887027968s) [noise=true , address=77.90.31.19:4856 , piece_id=XSMU5D6S5R6IAKHW4YCMW3FKE52XJG5EOXGDSBL247Y7GAF2L5YA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:21.684763342 (516h10m21.555846377s) [noise=true , address=145.79.227.35:4510 , piece_id=KCQTGDBXIE63VYMPKIIHAGPCYVYUTCEDROTO5JHQC2LKPJC5Z4WA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:22.446094653 (516h10m22.317177698s) [noise=true , address=77.90.31.18:4813 , piece_id=TUC6467NDUILBNIAZ6DROVTUB4CRWZALOQZUVIEJHHQYPBLOGGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:24.365452157 (516h10m24.236535202s) [noise=true , address=145.79.227.24:4615 , piece_id=HMELW7RWQ67I37Y5KXFL2UKXZMH434U25WWFZVYATGKI2BSNZLZA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:31.513341529 (516h10m31.384424564s) [noise=true , address=145.79.227.34:4909 , piece_id=RQKUC5AYJ76AABCIAXETYIW44F7X4KUFHIRCSYASRDJPMVVBCQ3A, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:32.290856236 (516h10m32.161939301s) [noise=true , address=145.79.227.10:4423 , piece_id=VCD3WLQ3Y7Z5YGFZFBPPLXAW35HEYECCJ6G4PSFCOMXK45Z2XC5Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578538275 (516h10m35.44962131s) [noise=true , address=145.79.227.44:4198 , piece_id=PKPSSATYA2U7RZ7TXVRK42VEP3L4TZU5EFFLNAVKNOVBRBL5RQHA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: <nil> Dec 27 03:39:35.578557731 (516h10m35.449640766s) [seg_pos={0 0}, upload=452901]: Segment reached optimal threshold of 14 pieces. Dec 27 03:39:35.578728707 (516h10m35.449811742s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF Dec 27 03:39:35.578734489 (516h10m35.449817524s) [noise=true , address=145.79.227.49:4353 , piece_id=3GKA4YD7YJCL6W6XXOLXWYCKNTQQDZE3QIXLDBVV5BBOOO5R6S3Q, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.578899161 (516h10m35.449982207s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo): protocol: rpc: dial tcp 145.79.227.29:4949: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.578904221 (516h10m35.449987266s) [noise=true , address=145.79.227.29:4949 , piece_id=33M6FWKPS6RV2MDNA4ZQT77GSHBY3LQDGPQ2UKJHIE54YPF377AA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579127447 (516h10m35.450210482s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: read tcp 188.166.225.170:58082->145.79.227.25:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579135101 (516h10m35.450218136s) [noise=true , address=145.79.227.25:4637 , piece_id=45KVP3ODHATEH7WXYG3SLYH44JLW2X4JKWD46EITGL64FFOEJGVA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579158115 (516h10m35.45024115s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: read tcp 188.166.225.170:40188->145.79.227.40:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:35.579162754 (516h10m35.450245799s) [noise=true , address=145.79.227.40:4064 , piece_id=2PN5ESPL7XICG73GVFV2W5WTC4WI53CWD3YSD2QVQQALVU3FVKFA, seg_pos={0 0}, upload=452901]: Long tail cancellation due to optimal threshold Dec 27 03:39:35.579173815 (516h10m35.45025686s) [seg_pos={0 0}, upload=452901]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 03:39:35.579230252 (516h10m35.450313287s) [upload=452901]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:39:35.579234711 (516h10m35.450317746s) [upload=452901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:39:35.873139933 (516h10m35.744222978s) [upload=452901]: Done uploading segments... Dec 27 03:39:35.873157767 (516h10m35.744240802s) [upload=452901]: Upload finished. err: <nil> Duration: 1m0.003601229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXXXX.] duration: 6.446455817s trace: https://jaeger.dev.storj.io/trace/55665fd13b621f5f FAIL (2025-12-27 03:40:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Dec 27 03:39:26.985105709 (516h10m26.856188775s) [upload=452906]: Starting upload Dec 27 03:39:26.985203556 (516h10m26.856286601s) [upload=452906]: Done starting upload Dec 27 03:39:26.985260815 (516h10m26.85634388s) [upload=452906]: Uploading segments... Dec 27 03:39:26.985472057 (516h10m26.856555123s) [upload=452906]: Got next segment. Inline: false Dec 27 03:39:26.985608527 (516h10m26.856691572s) [upload=452906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:39:26.985616001 (516h10m26.856699036s) [upload=452906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:39:27.234377962 (516h10m27.105460997s) [seg_pos={0 0}, upload=452906]: Begin upload segment... Dec 27 03:39:27.234582170 (516h10m27.105665205s) [seg_pos={0 0}, upload=452906]: Done begin upload segment. Dec 27 03:39:27.234664918 (516h10m27.105747963s) [noise=true , address=145.79.227.19:4079 , piece_id=3GCTAKIZJZIMR5GU2MQR2KAZVXOFWA3XIRCZNF2XFHKQFJPS7OQQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.234676039 (516h10m27.105759074s) [noise=true , address=145.79.227.3:4149 , piece_id=BXC3JCN7UCXD5OUM6CRMG3JKJAK4MRANV3TNFDCQIK6VF3RL2FQA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.235361825 (516h10m27.10644488s) [noise=true , address=145.79.227.42:4121 , piece_id=F2UUD3FS42AJLA7G5ZNHQNHZG5HUQV67IFJXVADQW45SLA7HNDWQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.235793016 (516h10m27.106876051s) [noise=true , address=45.91.116.7:4009 , piece_id=S6ZGJ57C6KNPGKJ6KKJFULGEXOJHF3V5CXKH7OMP3I6XV24W3OOA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.236136410 (516h10m27.107219445s) [noise=true , address=77.90.31.13:4627 , piece_id=UUGSAOI35FYF3GTYOLPK45ZKITGWRNP5S6TVCAWCA7KYK4LVH6SQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.236600674 (516h10m27.107683709s) [noise=true , address=145.79.227.47:4266 , piece_id=Y5N5RQ7IL7AOLW7A3O4F3FI7IW25TZ4Q5Q4MGR3I4FKQHX6CCX6A, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.237021356 (516h10m27.108104401s) [noise=true , address=145.79.227.48:4303 , piece_id=NHSMT3B75MKI4KT6HQ7SHVFMILK2KTGJ35FHPDC2NFP3GDQ76XKA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.237387392 (516h10m27.108470437s) [noise=true , address=145.79.227.6:4257 , piece_id=DVVAON7RANMKHB4524MQ7RR3JYJ2PZDDIECOWDDDOCLFPIYDBERA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.238262909 (516h10m27.109345954s) [noise=true , address=45.91.116.6:4974 , piece_id=7OWB4QHGWLRZR52EOETRIWPQHJLAKQ454XGAB3SDMORHMTGSDQRA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.241114659 (516h10m27.112197704s) [noise=true , address=145.79.227.46:4603 , piece_id=YDLJCVFNLEJTMLZQ7IENE5OQV463C7LTKV4IAJSMXRNJZEDQGPLQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.241307838 (516h10m27.112390893s) [noise=true , address=145.79.227.52:4446 , piece_id=RMQVT4J7VJ6CANBGZQZJTYZK5CV7LMBPTWA3ZQ2M6LWJKS252AKA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.244289434 (516h10m27.11537251s) [noise=true , address=45.91.116.5:4298 , piece_id=PXVP77HW444ID2UZMHXG4VXJHWEFGBYOJVBGFFS77DZBTAMKHU6Q, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.246396126 (516h10m27.117479171s) [noise=true , address=45.91.116.11:4120 , piece_id=7Z3HZHB4BDWOIRYOVABGGRDFBDR2YR4BV5ET3WSCVIURFSBOJRFQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.247828444 (516h10m27.118911479s) [noise=true , address=45.91.116.14:4190 , piece_id=5YMFHTUCEQSYEB3SDTUYNWXCUE7IL6LZSS3ILDG4VHGPQGPTGRNQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.253181397 (516h10m27.124264432s) [noise=true , address=45.91.116.9:4264 , piece_id=FMNX2WJV5QOWTESP6BJGLQNZXMU73SHTSDZVADORLPJYEX7LUIUQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.255433394 (516h10m27.126516439s) [noise=true , address=45.91.116.8:4048 , piece_id=CGLOUBBELU6RUMFKIZPLGKO4G4HQBO2RZAGQAPC4IF6HRFDAMRAA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:29.160963098 (516h10m29.032046133s) [upload=452906]: Next returned nil segment Dec 27 03:39:29.160980011 (516h10m29.032063056s) [upload=452906]: Waiting for error group managing segments... Dec 27 03:39:29.464416061 (516h10m29.335499096s) [noise=true , address=77.90.31.13:4627 , piece_id=UUGSAOI35FYF3GTYOLPK45ZKITGWRNP5S6TVCAWCA7KYK4LVH6SQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.485074178 (516h10m29.356157213s) [noise=true , address=45.91.116.6:4974 , piece_id=7OWB4QHGWLRZR52EOETRIWPQHJLAKQ454XGAB3SDMORHMTGSDQRA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.521671727 (516h10m29.392754762s) [noise=true , address=45.91.116.14:4190 , piece_id=5YMFHTUCEQSYEB3SDTUYNWXCUE7IL6LZSS3ILDG4VHGPQGPTGRNQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.533632793 (516h10m29.404715838s) [noise=true , address=45.91.116.7:4009 , piece_id=S6ZGJ57C6KNPGKJ6KKJFULGEXOJHF3V5CXKH7OMP3I6XV24W3OOA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.536599642 (516h10m29.407682677s) [noise=true , address=45.91.116.8:4048 , piece_id=CGLOUBBELU6RUMFKIZPLGKO4G4HQBO2RZAGQAPC4IF6HRFDAMRAA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.563367062 (516h10m29.434450097s) [noise=true , address=45.91.116.5:4298 , piece_id=PXVP77HW444ID2UZMHXG4VXJHWEFGBYOJVBGFFS77DZBTAMKHU6Q, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.605105059 (516h10m29.476188094s) [noise=true , address=45.91.116.11:4120 , piece_id=7Z3HZHB4BDWOIRYOVABGGRDFBDR2YR4BV5ET3WSCVIURFSBOJRFQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.747878658 (516h10m29.618961703s) [noise=true , address=45.91.116.9:4264 , piece_id=FMNX2WJV5QOWTESP6BJGLQNZXMU73SHTSDZVADORLPJYEX7LUIUQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:34.637162203 (516h10m34.508245248s) [noise=true , address=145.79.227.46:4603 , piece_id=YDLJCVFNLEJTMLZQ7IENE5OQV463C7LTKV4IAJSMXRNJZEDQGPLQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:49.648026810 (516h10m49.519109845s) [noise=true , address=145.79.227.48:4303 , piece_id=NHSMT3B75MKI4KT6HQ7SHVFMILK2KTGJ35FHPDC2NFP3GDQ76XKA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:56.298573886 (516h10m56.169656921s) [noise=true , address=145.79.227.42:4121 , piece_id=F2UUD3FS42AJLA7G5ZNHQNHZG5HUQV67IFJXVADQW45SLA7HNDWQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:58258->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:58258->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58258->145.79.227.42:4121: use of closed network connection Dec 27 03:39:56.298700857 (516h10m56.169783892s) [noise=true , address=145.79.227.19:4079 , piece_id=3GCTAKIZJZIMR5GU2MQR2KAZVXOFWA3XIRCZNF2XFHKQFJPS7OQQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: expected piece hash; context deadline exceeded; EOF Dec 27 03:39:56.298731406 (516h10m56.169814441s) [noise=true , address=145.79.227.52:4446 , piece_id=RMQVT4J7VJ6CANBGZQZJTYZK5CV7LMBPTWA3ZQ2M6LWJKS252AKA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:145.79.227.52:4446): protocol: rpc: dial tcp 145.79.227.52:4446: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:56.298880098 (516h10m56.169963133s) [noise=true , address=145.79.227.3:4149 , piece_id=BXC3JCN7UCXD5OUM6CRMG3JKJAK4MRANV3TNFDCQIK6VF3RL2FQA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:35346->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:56.299156855 (516h10m56.170239911s) [noise=true , address=145.79.227.47:4266 , piece_id=Y5N5RQ7IL7AOLW7A3O4F3FI7IW25TZ4Q5Q4MGR3I4FKQHX6CCX6A, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, address:145.79.227.47:4266): protocol: read tcp 188.166.225.170:51054->145.79.227.47:4266: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:56.299356496 (516h10m56.170439531s) [noise=true , address=145.79.227.6:4257 , piece_id=DVVAON7RANMKHB4524MQ7RR3JYJ2PZDDIECOWDDDOCLFPIYDBERA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:49656->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:56.299387515 (516h10m56.17047055s) [seg_pos={0 0}, upload=452906]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Dec 27 03:39:56.537434753 (516h10m56.408517788s) [upload=452906]: Done uploading segments... Dec 27 03:39:56.537495759 (516h10m56.408578794s) [upload=452906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.446488759s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXXXX] duration: 30.241284486s trace: https://jaeger.dev.storj.io/trace/55665fd13b621f5f FAIL (2025-12-27 03:39:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Dec 27 03:39:26.985105709 (516h10m26.856188775s) [upload=452906]: Starting upload Dec 27 03:39:26.985203556 (516h10m26.856286601s) [upload=452906]: Done starting upload Dec 27 03:39:26.985260815 (516h10m26.85634388s) [upload=452906]: Uploading segments... Dec 27 03:39:26.985472057 (516h10m26.856555123s) [upload=452906]: Got next segment. Inline: false Dec 27 03:39:26.985608527 (516h10m26.856691572s) [upload=452906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:39:26.985616001 (516h10m26.856699036s) [upload=452906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:39:27.234377962 (516h10m27.105460997s) [seg_pos={0 0}, upload=452906]: Begin upload segment... Dec 27 03:39:27.234582170 (516h10m27.105665205s) [seg_pos={0 0}, upload=452906]: Done begin upload segment. Dec 27 03:39:27.234664918 (516h10m27.105747963s) [noise=true , address=145.79.227.19:4079 , piece_id=3GCTAKIZJZIMR5GU2MQR2KAZVXOFWA3XIRCZNF2XFHKQFJPS7OQQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.234676039 (516h10m27.105759074s) [noise=true , address=145.79.227.3:4149 , piece_id=BXC3JCN7UCXD5OUM6CRMG3JKJAK4MRANV3TNFDCQIK6VF3RL2FQA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.235361825 (516h10m27.10644488s) [noise=true , address=145.79.227.42:4121 , piece_id=F2UUD3FS42AJLA7G5ZNHQNHZG5HUQV67IFJXVADQW45SLA7HNDWQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.235793016 (516h10m27.106876051s) [noise=true , address=45.91.116.7:4009 , piece_id=S6ZGJ57C6KNPGKJ6KKJFULGEXOJHF3V5CXKH7OMP3I6XV24W3OOA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.236136410 (516h10m27.107219445s) [noise=true , address=77.90.31.13:4627 , piece_id=UUGSAOI35FYF3GTYOLPK45ZKITGWRNP5S6TVCAWCA7KYK4LVH6SQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.236600674 (516h10m27.107683709s) [noise=true , address=145.79.227.47:4266 , piece_id=Y5N5RQ7IL7AOLW7A3O4F3FI7IW25TZ4Q5Q4MGR3I4FKQHX6CCX6A, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.237021356 (516h10m27.108104401s) [noise=true , address=145.79.227.48:4303 , piece_id=NHSMT3B75MKI4KT6HQ7SHVFMILK2KTGJ35FHPDC2NFP3GDQ76XKA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.237387392 (516h10m27.108470437s) [noise=true , address=145.79.227.6:4257 , piece_id=DVVAON7RANMKHB4524MQ7RR3JYJ2PZDDIECOWDDDOCLFPIYDBERA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.238262909 (516h10m27.109345954s) [noise=true , address=45.91.116.6:4974 , piece_id=7OWB4QHGWLRZR52EOETRIWPQHJLAKQ454XGAB3SDMORHMTGSDQRA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.241114659 (516h10m27.112197704s) [noise=true , address=145.79.227.46:4603 , piece_id=YDLJCVFNLEJTMLZQ7IENE5OQV463C7LTKV4IAJSMXRNJZEDQGPLQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.241307838 (516h10m27.112390893s) [noise=true , address=145.79.227.52:4446 , piece_id=RMQVT4J7VJ6CANBGZQZJTYZK5CV7LMBPTWA3ZQ2M6LWJKS252AKA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.244289434 (516h10m27.11537251s) [noise=true , address=45.91.116.5:4298 , piece_id=PXVP77HW444ID2UZMHXG4VXJHWEFGBYOJVBGFFS77DZBTAMKHU6Q, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.246396126 (516h10m27.117479171s) [noise=true , address=45.91.116.11:4120 , piece_id=7Z3HZHB4BDWOIRYOVABGGRDFBDR2YR4BV5ET3WSCVIURFSBOJRFQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.247828444 (516h10m27.118911479s) [noise=true , address=45.91.116.14:4190 , piece_id=5YMFHTUCEQSYEB3SDTUYNWXCUE7IL6LZSS3ILDG4VHGPQGPTGRNQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.253181397 (516h10m27.124264432s) [noise=true , address=45.91.116.9:4264 , piece_id=FMNX2WJV5QOWTESP6BJGLQNZXMU73SHTSDZVADORLPJYEX7LUIUQ, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:27.255433394 (516h10m27.126516439s) [noise=true , address=45.91.116.8:4048 , piece_id=CGLOUBBELU6RUMFKIZPLGKO4G4HQBO2RZAGQAPC4IF6HRFDAMRAA, seg_pos={0 0}, upload=452906]: Uploading piece... Dec 27 03:39:29.160963098 (516h10m29.032046133s) [upload=452906]: Next returned nil segment Dec 27 03:39:29.160980011 (516h10m29.032063056s) [upload=452906]: Waiting for error group managing segments... Dec 27 03:39:29.464416061 (516h10m29.335499096s) [noise=true , address=77.90.31.13:4627 , piece_id=UUGSAOI35FYF3GTYOLPK45ZKITGWRNP5S6TVCAWCA7KYK4LVH6SQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.485074178 (516h10m29.356157213s) [noise=true , address=45.91.116.6:4974 , piece_id=7OWB4QHGWLRZR52EOETRIWPQHJLAKQ454XGAB3SDMORHMTGSDQRA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.521671727 (516h10m29.392754762s) [noise=true , address=45.91.116.14:4190 , piece_id=5YMFHTUCEQSYEB3SDTUYNWXCUE7IL6LZSS3ILDG4VHGPQGPTGRNQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.533632793 (516h10m29.404715838s) [noise=true , address=45.91.116.7:4009 , piece_id=S6ZGJ57C6KNPGKJ6KKJFULGEXOJHF3V5CXKH7OMP3I6XV24W3OOA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.536599642 (516h10m29.407682677s) [noise=true , address=45.91.116.8:4048 , piece_id=CGLOUBBELU6RUMFKIZPLGKO4G4HQBO2RZAGQAPC4IF6HRFDAMRAA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.563367062 (516h10m29.434450097s) [noise=true , address=45.91.116.5:4298 , piece_id=PXVP77HW444ID2UZMHXG4VXJHWEFGBYOJVBGFFS77DZBTAMKHU6Q, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.605105059 (516h10m29.476188094s) [noise=true , address=45.91.116.11:4120 , piece_id=7Z3HZHB4BDWOIRYOVABGGRDFBDR2YR4BV5ET3WSCVIURFSBOJRFQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:29.747878658 (516h10m29.618961703s) [noise=true , address=45.91.116.9:4264 , piece_id=FMNX2WJV5QOWTESP6BJGLQNZXMU73SHTSDZVADORLPJYEX7LUIUQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:34.637162203 (516h10m34.508245248s) [noise=true , address=145.79.227.46:4603 , piece_id=YDLJCVFNLEJTMLZQ7IENE5OQV463C7LTKV4IAJSMXRNJZEDQGPLQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:49.648026810 (516h10m49.519109845s) [noise=true , address=145.79.227.48:4303 , piece_id=NHSMT3B75MKI4KT6HQ7SHVFMILK2KTGJ35FHPDC2NFP3GDQ76XKA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: <nil> Dec 27 03:39:56.298573886 (516h10m56.169656921s) [noise=true , address=145.79.227.42:4121 , piece_id=F2UUD3FS42AJLA7G5ZNHQNHZG5HUQV67IFJXVADQW45SLA7HNDWQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:58258->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:58258->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58258->145.79.227.42:4121: use of closed network connection Dec 27 03:39:56.298700857 (516h10m56.169783892s) [noise=true , address=145.79.227.19:4079 , piece_id=3GCTAKIZJZIMR5GU2MQR2KAZVXOFWA3XIRCZNF2XFHKQFJPS7OQQ, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: expected piece hash; context deadline exceeded; EOF Dec 27 03:39:56.298731406 (516h10m56.169814441s) [noise=true , address=145.79.227.52:4446 , piece_id=RMQVT4J7VJ6CANBGZQZJTYZK5CV7LMBPTWA3ZQ2M6LWJKS252AKA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:145.79.227.52:4446): protocol: rpc: dial tcp 145.79.227.52:4446: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:56.298880098 (516h10m56.169963133s) [noise=true , address=145.79.227.3:4149 , piece_id=BXC3JCN7UCXD5OUM6CRMG3JKJAK4MRANV3TNFDCQIK6VF3RL2FQA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:35346->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:56.299156855 (516h10m56.170239911s) [noise=true , address=145.79.227.47:4266 , piece_id=Y5N5RQ7IL7AOLW7A3O4F3FI7IW25TZ4Q5Q4MGR3I4FKQHX6CCX6A, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, address:145.79.227.47:4266): protocol: read tcp 188.166.225.170:51054->145.79.227.47:4266: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:56.299356496 (516h10m56.170439531s) [noise=true , address=145.79.227.6:4257 , piece_id=DVVAON7RANMKHB4524MQ7RR3JYJ2PZDDIECOWDDDOCLFPIYDBERA, seg_pos={0 0}, upload=452906]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:49656->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:39:56.299387515 (516h10m56.17047055s) [seg_pos={0 0}, upload=452906]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Dec 27 03:39:56.537434753 (516h10m56.408517788s) [upload=452906]: Done uploading segments... Dec 27 03:39:56.537495759 (516h10m56.408578794s) [upload=452906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.241306036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXXX] duration: 30.003518058s trace: https://jaeger.dev.storj.io/trace/2ed0d17bd22d0815 FAIL (2025-12-27 03:39:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 03:38:55.759320021 (516h9m55.630403056s) [upload=452897]: Starting upload Dec 27 03:38:55.759398119 (516h9m55.630481154s) [upload=452897]: Done starting upload Dec 27 03:38:55.759435190 (516h9m55.630518225s) [upload=452897]: Uploading segments... Dec 27 03:38:55.759679234 (516h9m55.63076229s) [upload=452897]: Got next segment. Inline: false Dec 27 03:38:55.759734530 (516h9m55.630817575s) [upload=452897]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:38:55.759739690 (516h9m55.630822725s) [upload=452897]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:38:56.008798796 (516h9m55.879881831s) [seg_pos={0 0}, upload=452897]: Begin upload segment... Dec 27 03:38:56.008842500 (516h9m55.879925535s) [seg_pos={0 0}, upload=452897]: Done begin upload segment. Dec 27 03:38:56.008869350 (516h9m55.879952396s) [noise=true , address=145.79.227.48:4303 , piece_id=7A2JQDIV7X3D7XIAAJOXPV3IFPOVUFTELMBFANYJVLNMEZOYROUQ, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.008913435 (516h9m55.87999647s) [noise=true , address=145.79.227.11:4730 , piece_id=LXIE3ID23UZIRGP76OSIDLY3YQYFJQUY67C4RRD6WJT733YGC6VA, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.010544731 (516h9m55.881627776s) [noise=true , address=145.79.227.2:4115 , piece_id=G3NK3DHFSUOOMXLGLCH5GOSKYTUJDZTVNTXMY3LL4C3DULWN7UUA, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.010937328 (516h9m55.882020404s) [noise=true , address=145.79.227.16:4911 , piece_id=EBR22PF7LHTHNHEMHR47RXNDMUTDWIU6SQ2ARTT7TKHZDXMUG7DQ, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.011311842 (516h9m55.882394887s) [noise=true , address=45.91.116.11:4117 , piece_id=XG6KNEMBCAZYMXESEEOEB7ZJJZCREPWINN3LWY76Y3HWCWRFFS3A, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.011902036 (516h9m55.882985071s) [noise=true , address=145.79.227.47:4266 , piece_id=XUKR2VELI7QYKMRXYAMYY6CI62ZJUFBCCRHQPBFARLNUBALYETWA, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.012295615 (516h9m55.88337866s) [noise=true , address=145.79.227.51:4409 , piece_id=HOTFLCVQTCD54D4GRANYDW5D6AJIV3AUVYW4HZX5SWJP3B35YOZQ, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.012644549 (516h9m55.883727584s) [noise=true , address=77.90.31.19:4841 , piece_id=Y4NGM4UG5T66L57HXNWSCZMKGEKDSOL6Y3YGXKODDMZJRQ32HO7Q, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.013049601 (516h9m55.884132646s) [noise=true , address=45.91.116.14:4189 , piece_id=HOLS5ASQDTD2K7JRGVXSBIGQ4DGHT4ZZ3RUV3BSUYYPCHG25RAIA, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.014401095 (516h9m55.88548413s) [noise=true , address=145.79.227.9:4366 , piece_id=7QYXQKSR7YI5MH2WFYQNKABVUZR2JUX7KV7CZIGOEZOLKCQSSMKA, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.014496486 (516h9m55.885579531s) [noise=true , address=77.90.31.21:4877 , piece_id=43ZVU23QLW5IVLFUGQHD2EWKMTPHFQ4SESAO5IHHL3HECI3Q2LZQ, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.016603368 (516h9m55.887686413s) [noise=true , address=145.79.227.28:4725 , piece_id=QBOV2JIJ3FL6YE47WJCNK4RKQ6PQJCSFJXK5PUN2VH5MLUMWW3NA, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.017904596 (516h9m55.888987641s) [noise=true , address=45.91.116.5:4298 , piece_id=YK6NP3JBK6RAB4OQHMYX6IM5LTWCCGYAONHFSSCPSTJDSWCZJADQ, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.019061278 (516h9m55.890144313s) [noise=true , address=77.90.31.18:4808 , piece_id=LKLYUGBVSNDHQUP23YACQYMGIG2L6MKFFZXIHBVCMTZUKZHRFVXQ, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.020143339 (516h9m55.891226384s) [noise=true , address=77.90.31.14:4664 , piece_id=V7MAAOB7PWBDT3HEVJBDXVZADWRODKD7DLOMNLEBD5VBBJTO3ITA, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:56.021310290 (516h9m55.892393335s) [noise=true , address=145.79.227.42:4122 , piece_id=L4UDL5BEAVFCYEWHSWTAQTWOJRQQ2REY2HF23VOOEXTFYB3IIB5Q, seg_pos={0 0}, upload=452897]: Uploading piece... Dec 27 03:38:57.730627348 (516h9m57.601710383s) [upload=452897]: Next returned nil segment Dec 27 03:38:57.730643960 (516h9m57.601726995s) [upload=452897]: Waiting for error group managing segments... Dec 27 03:38:58.010579541 (516h9m57.881662576s) [noise=true , address=145.79.227.11:4730 , piece_id=LXIE3ID23UZIRGP76OSIDLY3YQYFJQUY67C4RRD6WJT733YGC6VA, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.017167966 (516h9m57.888251011s) [noise=true , address=145.79.227.2:4115 , piece_id=G3NK3DHFSUOOMXLGLCH5GOSKYTUJDZTVNTXMY3LL4C3DULWN7UUA, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.019796392 (516h9m57.890879437s) [noise=true , address=145.79.227.16:4911 , piece_id=EBR22PF7LHTHNHEMHR47RXNDMUTDWIU6SQ2ARTT7TKHZDXMUG7DQ, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.030802068 (516h9m57.901885103s) [noise=true , address=145.79.227.28:4725 , piece_id=QBOV2JIJ3FL6YE47WJCNK4RKQ6PQJCSFJXK5PUN2VH5MLUMWW3NA, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.031110586 (516h9m57.902193631s) [noise=true , address=145.79.227.48:4303 , piece_id=7A2JQDIV7X3D7XIAAJOXPV3IFPOVUFTELMBFANYJVLNMEZOYROUQ, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.033099573 (516h9m57.904182608s) [noise=true , address=145.79.227.42:4122 , piece_id=L4UDL5BEAVFCYEWHSWTAQTWOJRQQ2REY2HF23VOOEXTFYB3IIB5Q, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.040915036 (516h9m57.911998081s) [noise=true , address=145.79.227.9:4366 , piece_id=7QYXQKSR7YI5MH2WFYQNKABVUZR2JUX7KV7CZIGOEZOLKCQSSMKA, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.044249093 (516h9m57.915332139s) [noise=true , address=145.79.227.51:4409 , piece_id=HOTFLCVQTCD54D4GRANYDW5D6AJIV3AUVYW4HZX5SWJP3B35YOZQ, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.050336526 (516h9m57.921419561s) [noise=true , address=145.79.227.47:4266 , piece_id=XUKR2VELI7QYKMRXYAMYY6CI62ZJUFBCCRHQPBFARLNUBALYETWA, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.194815984 (516h9m58.065899029s) [noise=true , address=77.90.31.18:4808 , piece_id=LKLYUGBVSNDHQUP23YACQYMGIG2L6MKFFZXIHBVCMTZUKZHRFVXQ, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.243066180 (516h9m58.114149225s) [noise=true , address=45.91.116.11:4117 , piece_id=XG6KNEMBCAZYMXESEEOEB7ZJJZCREPWINN3LWY76Y3HWCWRFFS3A, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.291879291 (516h9m58.162962346s) [noise=true , address=45.91.116.14:4189 , piece_id=HOLS5ASQDTD2K7JRGVXSBIGQ4DGHT4ZZ3RUV3BSUYYPCHG25RAIA, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.307228939 (516h9m58.178311974s) [noise=true , address=77.90.31.19:4841 , piece_id=Y4NGM4UG5T66L57HXNWSCZMKGEKDSOL6Y3YGXKODDMZJRQ32HO7Q, seg_pos={0 0}, upload=452897]: Done uploading piece. err: <nil> Dec 27 03:38:58.307252684 (516h9m58.17833572s) [seg_pos={0 0}, upload=452897]: Segment reached optimal threshold of 13 pieces. Dec 27 03:38:58.307421435 (516h9m58.17850447s) [noise=true , address=45.91.116.5:4298 , piece_id=YK6NP3JBK6RAB4OQHMYX6IM5LTWCCGYAONHFSSCPSTJDSWCZJADQ, seg_pos={0 0}, upload=452897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 03:38:58.307429742 (516h9m58.178512777s) [noise=true , address=45.91.116.5:4298 , piece_id=YK6NP3JBK6RAB4OQHMYX6IM5LTWCCGYAONHFSSCPSTJDSWCZJADQ, seg_pos={0 0}, upload=452897]: Long tail cancellation due to optimal threshold Dec 27 03:38:58.307584866 (516h9m58.178667922s) [noise=true , address=77.90.31.21:4877 , piece_id=43ZVU23QLW5IVLFUGQHD2EWKMTPHFQ4SESAO5IHHL3HECI3Q2LZQ, seg_pos={0 0}, upload=452897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 03:38:58.307595096 (516h9m58.178678131s) [noise=true , address=77.90.31.21:4877 , piece_id=43ZVU23QLW5IVLFUGQHD2EWKMTPHFQ4SESAO5IHHL3HECI3Q2LZQ, seg_pos={0 0}, upload=452897]: Long tail cancellation due to optimal threshold Dec 27 03:38:58.307616146 (516h9m58.178699181s) [noise=true , address=77.90.31.14:4664 , piece_id=V7MAAOB7PWBDT3HEVJBDXVZADWRODKD7DLOMNLEBD5VBBJTO3ITA, seg_pos={0 0}, upload=452897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 27 03:38:58.307622057 (516h9m58.178705092s) [noise=true , address=77.90.31.14:4664 , piece_id=V7MAAOB7PWBDT3HEVJBDXVZADWRODKD7DLOMNLEBD5VBBJTO3ITA, seg_pos={0 0}, upload=452897]: Long tail cancellation due to optimal threshold Dec 27 03:38:58.307639751 (516h9m58.178722796s) [seg_pos={0 0}, upload=452897]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:38:58.307729101 (516h9m58.178812136s) [upload=452897]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:38:58.307736867 (516h9m58.178819902s) [upload=452897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:38:58.606198044 (516h9m58.477281079s) [upload=452897]: Done uploading segments... Dec 27 03:38:58.606221249 (516h9m58.477304294s) [upload=452897]: Upload finished. err: <nil> Duration: 30.003540069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXXX] duration: 30.004336599s trace: https://jaeger.dev.storj.io/trace/46bd4ea2f8f4d066 FAIL (2025-12-27 03:38:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e starting upload Dec 27 03:38:00.240423400 (516h9m0.111506435s) [upload=452884]: Uploading segments... Dec 27 03:38:00.240507409 (516h9m0.111590454s) [upload=452884]: Got next segment. Inline: false Dec 27 03:38:00.240542225 (516h9m0.11162526s) [upload=452884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:38:00.240546684 (516h9m0.111629719s) [upload=452884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:38:00.480870337 (516h9m0.351953372s) [seg_pos={0 0}, upload=452884]: Begin upload segment... Dec 27 03:38:00.480923888 (516h9m0.352006923s) [seg_pos={0 0}, upload=452884]: Done begin upload segment. Dec 27 03:38:00.480968914 (516h9m0.352051949s) [noise=true , address=145.79.227.9:4366 , piece_id=QD3RMBQJ6CXQ7FVPUA7ZDQ4V4562MXLBU3LIIR5HSSDJIZTRGDUA, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.481012447 (516h9m0.352095482s) [noise=true , address=145.79.227.3:4151 , piece_id=CG4HHD6R5RUCXOZJSXNN3Z7IHYBZHO235GQSVTIV2PEJOLQ5IHTA, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.481587642 (516h9m0.352670677s) [noise=true , address=145.79.227.4:4185 , piece_id=ILTARUYFJZ3SMM6EWOGCISNP4MGLVKJV7W7VKSS42FTXTKAWTIVQ, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.482019034 (516h9m0.353102079s) [noise=true , address=145.79.227.8:4330 , piece_id=5HTIJV54UXFJU7UTXSCQXV4C5DOWJIYQOJPEUVKBAFAZKRNTA64A, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.482029454 (516h9m0.353112499s) [noise=true , address=45.91.116.5:4299 , piece_id=3AI6XK3DHWPUUOLWDCYPJXJS7V7QZJMM5QYTXOMX2WHO3Z4TF4TA, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.482390260 (516h9m0.353473305s) [noise=true , address=145.79.227.15:4873 , piece_id=JCE4UYSVECL4SOJLEDGLBLS7EGGYG6LJB5EO5H4ULB73FBXDXZCA, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.482659674 (516h9m0.353742719s) [noise=true , address=77.90.31.14:4661 , piece_id=AS2BHRYV4N3HQ5TEEZYK3FC2DAF2PQY5X6IT4NKR7LIQBUZBF33A, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.482845267 (516h9m0.353928522s) [noise=true , address=45.91.116.10:4084 , piece_id=STNOXLWVQFXDHI3PLEL7JGJXLYVGII6QT3WBX5EYIWQIH4YLDC3A, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.483111514 (516h9m0.354194549s) [noise=true , address=45.91.116.9:4262 , piece_id=XDF3OSDK35GPQN7SWZCHXEHU36WRQ2PIHD7JMV6VSXKBRX5VKIBQ, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.483287890 (516h9m0.354370935s) [noise=true , address=45.91.116.14:4191 , piece_id=D4L3C2V64TB5LKUX2D7GSU2IEVXYEEEM4IHR7NHSG5KJ4QNWXVYA, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.483719421 (516h9m0.354802466s) [noise=true , address=145.79.227.38:4979 , piece_id=GB5CNGQF7Z5DBKYYS22A4KD3UYIPK5U7ZPS6IN6NTKM2QDFKUIBA, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.483754227 (516h9m0.354837272s) [noise=true , address=45.91.116.8:4045 , piece_id=JDAUZXDADNWT5GSQLTSDQ2RWPAGPDYP644CLCRZJZ3X5RIDEUAZQ, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.484141025 (516h9m0.35522406s) [noise=true , address=145.79.227.49:4337 , piece_id=EYKAPYLZRARVSQFRZMMQBS6PDBUDPZW2T2NXQCDJAF6Q6TLWAD4A, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.484358288 (516h9m0.355441333s) [noise=true , address=145.79.227.10:4401 , piece_id=RZLDLT5SJ5G3D73DESCZIX5Z3QV5AWTFIK63QRQR7GHOJSS2NZNA, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.484518523 (516h9m0.355601568s) [noise=true , address=77.90.31.10:4519 , piece_id=XOLXDBW2QQNCXJ6FU3Y4EITI3KWZZMLRWWI3EHQQMYZB4J7KY3EA, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:00.484857559 (516h9m0.355940604s) [noise=true , address=45.91.116.12:4225 , piece_id=7623QTAQQEPMGSGM2HQIXOBYEKMCYKHHL477KWRH3ZBBVOD4ZR2Q, seg_pos={0 0}, upload=452884]: Uploading piece... Dec 27 03:38:02.197132477 (516h9m2.068215522s) [upload=452884]: Next returned nil segment Dec 27 03:38:02.197153798 (516h9m2.068236833s) [upload=452884]: Waiting for error group managing segments... Dec 27 03:38:02.488924786 (516h9m2.360007821s) [noise=true , address=145.79.227.3:4151 , piece_id=CG4HHD6R5RUCXOZJSXNN3Z7IHYBZHO235GQSVTIV2PEJOLQ5IHTA, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.489271506 (516h9m2.360354551s) [noise=true , address=145.79.227.9:4366 , piece_id=QD3RMBQJ6CXQ7FVPUA7ZDQ4V4562MXLBU3LIIR5HSSDJIZTRGDUA, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.489628066 (516h9m2.360711111s) [noise=true , address=145.79.227.38:4979 , piece_id=GB5CNGQF7Z5DBKYYS22A4KD3UYIPK5U7ZPS6IN6NTKM2QDFKUIBA, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.495521528 (516h9m2.366604573s) [noise=true , address=145.79.227.10:4401 , piece_id=RZLDLT5SJ5G3D73DESCZIX5Z3QV5AWTFIK63QRQR7GHOJSS2NZNA, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.499798562 (516h9m2.370881607s) [noise=true , address=145.79.227.8:4330 , piece_id=5HTIJV54UXFJU7UTXSCQXV4C5DOWJIYQOJPEUVKBAFAZKRNTA64A, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.503387777 (516h9m2.374470822s) [noise=true , address=145.79.227.15:4873 , piece_id=JCE4UYSVECL4SOJLEDGLBLS7EGGYG6LJB5EO5H4ULB73FBXDXZCA, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.504183181 (516h9m2.375266226s) [noise=true , address=145.79.227.4:4185 , piece_id=ILTARUYFJZ3SMM6EWOGCISNP4MGLVKJV7W7VKSS42FTXTKAWTIVQ, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.506526423 (516h9m2.377609468s) [noise=true , address=145.79.227.49:4337 , piece_id=EYKAPYLZRARVSQFRZMMQBS6PDBUDPZW2T2NXQCDJAF6Q6TLWAD4A, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.717709737 (516h9m2.588792782s) [noise=true , address=45.91.116.14:4191 , piece_id=D4L3C2V64TB5LKUX2D7GSU2IEVXYEEEM4IHR7NHSG5KJ4QNWXVYA, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.733268463 (516h9m2.604351508s) [noise=true , address=45.91.116.8:4045 , piece_id=JDAUZXDADNWT5GSQLTSDQ2RWPAGPDYP644CLCRZJZ3X5RIDEUAZQ, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.854728657 (516h9m2.725811692s) [noise=true , address=45.91.116.12:4225 , piece_id=7623QTAQQEPMGSGM2HQIXOBYEKMCYKHHL477KWRH3ZBBVOD4ZR2Q, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:02.903844473 (516h9m2.774927509s) [noise=true , address=45.91.116.9:4262 , piece_id=XDF3OSDK35GPQN7SWZCHXEHU36WRQ2PIHD7JMV6VSXKBRX5VKIBQ, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:03.019313288 (516h9m2.890396333s) [noise=true , address=45.91.116.5:4299 , piece_id=3AI6XK3DHWPUUOLWDCYPJXJS7V7QZJMM5QYTXOMX2WHO3Z4TF4TA, seg_pos={0 0}, upload=452884]: Done uploading piece. err: <nil> Dec 27 03:38:03.019333727 (516h9m2.890416772s) [seg_pos={0 0}, upload=452884]: Segment reached optimal threshold of 13 pieces. Dec 27 03:38:03.019530040 (516h9m2.890613136s) [noise=true , address=45.91.116.10:4084 , piece_id=STNOXLWVQFXDHI3PLEL7JGJXLYVGII6QT3WBX5EYIWQIH4YLDC3A, seg_pos={0 0}, upload=452884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:58012->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:58012->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58012->45.91.116.10:4084: use of closed network connection Dec 27 03:38:03.019540862 (516h9m2.890623917s) [noise=true , address=45.91.116.10:4084 , piece_id=STNOXLWVQFXDHI3PLEL7JGJXLYVGII6QT3WBX5EYIWQIH4YLDC3A, seg_pos={0 0}, upload=452884]: Long tail cancellation due to optimal threshold Dec 27 03:38:03.019606106 (516h9m2.890689161s) [noise=true , address=77.90.31.14:4661 , piece_id=AS2BHRYV4N3HQ5TEEZYK3FC2DAF2PQY5X6IT4NKR7LIQBUZBF33A, seg_pos={0 0}, upload=452884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 03:38:03.019618028 (516h9m2.890701074s) [noise=true , address=77.90.31.14:4661 , piece_id=AS2BHRYV4N3HQ5TEEZYK3FC2DAF2PQY5X6IT4NKR7LIQBUZBF33A, seg_pos={0 0}, upload=452884]: Long tail cancellation due to optimal threshold Dec 27 03:38:03.019684645 (516h9m2.890767691s) [noise=true , address=77.90.31.10:4519 , piece_id=XOLXDBW2QQNCXJ6FU3Y4EITI3KWZZMLRWWI3EHQQMYZB4J7KY3EA, seg_pos={0 0}, upload=452884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:48922->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:48922->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48922->77.90.31.10:4519: use of closed network connection Dec 27 03:38:03.019693793 (516h9m2.890776838s) [noise=true , address=77.90.31.10:4519 , piece_id=XOLXDBW2QQNCXJ6FU3Y4EITI3KWZZMLRWWI3EHQQMYZB4J7KY3EA, seg_pos={0 0}, upload=452884]: Long tail cancellation due to optimal threshold Dec 27 03:38:03.019788994 (516h9m2.890872039s) [seg_pos={0 0}, upload=452884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:38:03.019875639 (516h9m2.890958684s) [upload=452884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:38:03.019883875 (516h9m2.89096692s) [upload=452884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:38:03.313565951 (516h9m3.184648986s) [upload=452884]: Done uploading segments... Dec 27 03:38:03.313595818 (516h9m3.184678863s) [upload=452884]: Upload finished. err: <nil> Duration: 30.004355896s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXXX] duration: 30.002992778s trace: https://jaeger.dev.storj.io/trace/11ea14676196dc86 FAIL (2025-12-27 03:37:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: PAN52Z7WSLFUDZMI4S5AIWJL5DXK7KDAMW3JP4A3DODAFDGEEOBQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: BEDZGEWKWTBLE673R3IPD43H7GKDMECNFJHTIT5HYJMJ5CBLW32A) context deadline exceeded ======LOGS====== Dec 27 03:37:16.314022234 (516h8m16.185105269s) [upload=452875]: Starting upload Dec 27 03:37:16.314116383 (516h8m16.185199428s) [upload=452875]: Done starting upload Dec 27 03:37:16.314319771 (516h8m16.185402806s) [upload=452875]: Uploading segments... Dec 27 03:37:16.314585676 (516h8m16.185668721s) [upload=452875]: Got next segment. Inline: false Dec 27 03:37:16.314658966 (516h8m16.185742011s) [upload=452875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:37:16.314666711 (516h8m16.185749746s) [upload=452875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:37:16.563935937 (516h8m16.435018982s) [seg_pos={0 0}, upload=452875]: Begin upload segment... Dec 27 03:37:16.563989540 (516h8m16.435072595s) [seg_pos={0 0}, upload=452875]: Done begin upload segment. Dec 27 03:37:16.564020959 (516h8m16.435104044s) [noise=true , address=145.79.227.45:4229 , piece_id=GNTIKLQRCE4KFTMMBQ5T5XCEQHNV7SR5DX5GJJAB4NERMUUHXT3A, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.564038843 (516h8m16.435121898s) [noise=true , address=145.79.227.4:4185 , piece_id=3MOF4GC5FJ3LMWM6CHTKTDYQWPDTNTFEVPGGWZSKLE3IATNNGYHA, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.564694682 (516h8m16.435777727s) [noise=true , address=77.90.31.12:4590 , piece_id=6PF7X5KY6RUY3SUNSL6WHB5NZO2F3S62BFPVPRV32KY3B3UCHI3Q, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.565069496 (516h8m16.436152541s) [noise=true , address=145.79.227.11:4730 , piece_id=4PY62QFVZ4AU5YUEACWQMW525XMFVP4EJA6XXHO62V7BDILOYKPA, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.565098340 (516h8m16.436181385s) [noise=true , address=45.91.116.14:4192 , piece_id=PAN52Z7WSLFUDZMI4S5AIWJL5DXK7KDAMW3JP4A3DODAFDGEEOBQ, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.565402940 (516h8m16.436485975s) [noise=true , address=145.79.227.49:4337 , piece_id=3K27TY2Q4CYHND6OKNHMZFOTSCIJFD2AIAKAK3VOEU4C6OSTKCJA, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.565584806 (516h8m16.436667842s) [noise=true , address=45.91.116.11:4119 , piece_id=R3BWC7BKPYZJTLGORXNXNME7WRXQQDTK6MI4YMXRI4CAHDRQWL3A, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.565658147 (516h8m16.436741192s) [noise=true , address=77.90.31.21:4880 , piece_id=BEDZGEWKWTBLE673R3IPD43H7GKDMECNFJHTIT5HYJMJ5CBLW32A, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.565948379 (516h8m16.437031414s) [noise=true , address=145.79.227.9:4365 , piece_id=W255DWRE225J4JL44FFBBQPYLIUQVMWLT5TQZSULN7BBNJLY7BXQ, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.566080611 (516h8m16.437163646s) [noise=true , address=45.91.116.7:4012 , piece_id=F6QWTSU5TKN6GVJUKN43XRG6D5KX64Y6OE3IKNFT6UETBJJKUPDA, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.566288257 (516h8m16.437371292s) [noise=true , address=45.91.116.9:4263 , piece_id=O7GNQLQGOFQQ5YXSPNHNWAHBXDUXIXA6CY42UDRNCJLAFVEIF3YA, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.566688448 (516h8m16.437771483s) [noise=true , address=145.79.227.50:4374 , piece_id=RAIFRUHMSVBXN23YEJPPGW4FUSYZ37CPUHPNQRLO3J7OZZXG6VBQ, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.566721281 (516h8m16.437804316s) [noise=true , address=145.79.227.43:4159 , piece_id=SCSKMR5BBCOLFOASCUQNS3K4XVY7XIO3GOHWI7Q5S5H4KQTISYNQ, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.567145118 (516h8m16.438228153s) [noise=true , address=45.91.116.10:4082 , piece_id=JUKFNANKW7NK46KSBKG355A2BIOZ5IUN3EY5LIU4ILKRAS6VUY4A, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.567568053 (516h8m16.438651098s) [noise=true , address=77.90.31.10:4518 , piece_id=2AQTQFWQEV26BYU4NVKIO3H4GFNQDPGAHSH27BQDXEZYW45JCDFA, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:16.568294897 (516h8m16.439377932s) [noise=true , address=145.79.227.28:4725 , piece_id=UR4U2CG42G2WVEBYZSWLKL2XBMUAW4BYNXLLMXKISNX3GSTIO3JQ, seg_pos={0 0}, upload=452875]: Uploading piece... Dec 27 03:37:18.279888289 (516h8m18.150971334s) [upload=452875]: Next returned nil segment Dec 27 03:37:18.279928896 (516h8m18.151011941s) [upload=452875]: Waiting for error group managing segments... Dec 27 03:37:18.564529340 (516h8m18.435612385s) [noise=true , address=145.79.227.11:4730 , piece_id=4PY62QFVZ4AU5YUEACWQMW525XMFVP4EJA6XXHO62V7BDILOYKPA, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:18.569730715 (516h8m18.44081376s) [noise=true , address=145.79.227.9:4365 , piece_id=W255DWRE225J4JL44FFBBQPYLIUQVMWLT5TQZSULN7BBNJLY7BXQ, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:18.570159611 (516h8m18.441242646s) [noise=true , address=145.79.227.49:4337 , piece_id=3K27TY2Q4CYHND6OKNHMZFOTSCIJFD2AIAKAK3VOEU4C6OSTKCJA, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:18.574673947 (516h8m18.445756982s) [noise=true , address=145.79.227.50:4374 , piece_id=RAIFRUHMSVBXN23YEJPPGW4FUSYZ37CPUHPNQRLO3J7OZZXG6VBQ, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:18.584071803 (516h8m18.455154868s) [noise=true , address=145.79.227.28:4725 , piece_id=UR4U2CG42G2WVEBYZSWLKL2XBMUAW4BYNXLLMXKISNX3GSTIO3JQ, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:18.584792205 (516h8m18.45587524s) [noise=true , address=145.79.227.4:4185 , piece_id=3MOF4GC5FJ3LMWM6CHTKTDYQWPDTNTFEVPGGWZSKLE3IATNNGYHA, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:18.603481843 (516h8m18.474564878s) [noise=true , address=145.79.227.45:4229 , piece_id=GNTIKLQRCE4KFTMMBQ5T5XCEQHNV7SR5DX5GJJAB4NERMUUHXT3A, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:18.608169930 (516h8m18.479252965s) [noise=true , address=145.79.227.43:4159 , piece_id=SCSKMR5BBCOLFOASCUQNS3K4XVY7XIO3GOHWI7Q5S5H4KQTISYNQ, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:18.766116821 (516h8m18.637199856s) [noise=true , address=77.90.31.21:4880 , piece_id=BEDZGEWKWTBLE673R3IPD43H7GKDMECNFJHTIT5HYJMJ5CBLW32A, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:18.842701913 (516h8m18.713784948s) [noise=true , address=45.91.116.9:4263 , piece_id=O7GNQLQGOFQQ5YXSPNHNWAHBXDUXIXA6CY42UDRNCJLAFVEIF3YA, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:19.607620926 (516h8m19.478703962s) [noise=true , address=45.91.116.10:4082 , piece_id=JUKFNANKW7NK46KSBKG355A2BIOZ5IUN3EY5LIU4ILKRAS6VUY4A, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:19.679280168 (516h8m19.550363214s) [noise=true , address=45.91.116.11:4119 , piece_id=R3BWC7BKPYZJTLGORXNXNME7WRXQQDTK6MI4YMXRI4CAHDRQWL3A, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:19.714226664 (516h8m19.585309709s) [noise=true , address=45.91.116.14:4192 , piece_id=PAN52Z7WSLFUDZMI4S5AIWJL5DXK7KDAMW3JP4A3DODAFDGEEOBQ, seg_pos={0 0}, upload=452875]: Done uploading piece. err: <nil> Dec 27 03:37:19.714254206 (516h8m19.585337251s) [seg_pos={0 0}, upload=452875]: Segment reached optimal threshold of 13 pieces. Dec 27 03:37:19.714574075 (516h8m19.58565712s) [noise=true , address=45.91.116.7:4012 , piece_id=F6QWTSU5TKN6GVJUKN43XRG6D5KX64Y6OE3IKNFT6UETBJJKUPDA, seg_pos={0 0}, upload=452875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 03:37:19.714584405 (516h8m19.58566745s) [noise=true , address=45.91.116.7:4012 , piece_id=F6QWTSU5TKN6GVJUKN43XRG6D5KX64Y6OE3IKNFT6UETBJJKUPDA, seg_pos={0 0}, upload=452875]: Long tail cancellation due to optimal threshold Dec 27 03:37:19.714681650 (516h8m19.585764685s) [noise=true , address=77.90.31.12:4590 , piece_id=6PF7X5KY6RUY3SUNSL6WHB5NZO2F3S62BFPVPRV32KY3B3UCHI3Q, seg_pos={0 0}, upload=452875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 03:37:19.714705616 (516h8m19.585788661s) [noise=true , address=77.90.31.12:4590 , piece_id=6PF7X5KY6RUY3SUNSL6WHB5NZO2F3S62BFPVPRV32KY3B3UCHI3Q, seg_pos={0 0}, upload=452875]: Long tail cancellation due to optimal threshold Dec 27 03:37:19.714860410 (516h8m19.585943445s) [noise=true , address=77.90.31.10:4518 , piece_id=2AQTQFWQEV26BYU4NVKIO3H4GFNQDPGAHSH27BQDXEZYW45JCDFA, seg_pos={0 0}, upload=452875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 03:37:19.714866903 (516h8m19.585949939s) [noise=true , address=77.90.31.10:4518 , piece_id=2AQTQFWQEV26BYU4NVKIO3H4GFNQDPGAHSH27BQDXEZYW45JCDFA, seg_pos={0 0}, upload=452875]: Long tail cancellation due to optimal threshold Dec 27 03:37:19.714883645 (516h8m19.58596668s) [seg_pos={0 0}, upload=452875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:37:19.714967985 (516h8m19.58605102s) [upload=452875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:37:19.714972955 (516h8m19.58605599s) [upload=452875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:37:20.019684434 (516h8m19.890767479s) [upload=452875]: Done uploading segments... Dec 27 03:37:20.019706988 (516h8m19.890790023s) [upload=452875]: Upload finished. err: <nil> Duration: 30.003010553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XXX] duration: 30.002819742s trace: https://jaeger.dev.storj.io/trace/13142f6740593fc4 FAIL (2025-12-27 03:36:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 03:36:16.391594286 (516h7m16.262677321s) [upload=452863]: Starting upload Dec 27 03:36:16.391668487 (516h7m16.262751522s) [upload=452863]: Done starting upload Dec 27 03:36:16.391755954 (516h7m16.262838999s) [upload=452863]: Uploading segments... Dec 27 03:36:16.391948200 (516h7m16.263031235s) [upload=452863]: Got next segment. Inline: false Dec 27 03:36:16.391985841 (516h7m16.263068876s) [upload=452863]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:36:16.391990380 (516h7m16.263073415s) [upload=452863]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:36:16.635194757 (516h7m16.506277792s) [seg_pos={0 0}, upload=452863]: Begin upload segment... Dec 27 03:36:16.635251354 (516h7m16.50633439s) [seg_pos={0 0}, upload=452863]: Done begin upload segment. Dec 27 03:36:16.635274669 (516h7m16.506357704s) [noise=true , address=45.91.116.9:4262 , piece_id=IKBNJQWTAHCEIAU4EFDYMXVC3K7I7K2SACKULT2D6SCG6S7AWMEA, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.635350784 (516h7m16.506433819s) [noise=true , address=77.90.31.12:4590 , piece_id=Q4PN6ZOUE6F7P3NR67DQYXHU7VF2LPZML2HFMNUMSGC4LBYCV3YA, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.635928985 (516h7m16.50701203s) [noise=true , address=77.90.31.14:4661 , piece_id=ZXOOFBTBKHRXHX5M2OULO44ZIHOJJ2UBKUWOIVY3MNXG7PB7Y34Q, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.636237984 (516h7m16.507321029s) [noise=true , address=45.91.116.13:4154 , piece_id=RNCIZFP3HEWLBIKH44W4MVP4SXDTUPESWSGKI22IEJLEFCC7MV7A, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.636501626 (516h7m16.507584681s) [noise=true , address=145.79.227.34:4907 , piece_id=ASSTOYCTSFKC3MDPIAQIG2ON6AGQKHWZ7O2GM76ISKBJERPOX7VQ, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.636523397 (516h7m16.507606442s) [noise=true , address=145.79.227.15:4875 , piece_id=G22UQYJNIVIBG23GGN6LHF5JRWU5PRRMAWXWQN4VXICXMMDDAFCA, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.636771209 (516h7m16.507854244s) [noise=true , address=45.91.116.12:4226 , piece_id=2Z2DXYJZKOQAHJY4JLBDOYUKZ6LPTGZ6M2BHQMGVPK7RESMP36XA, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.637020895 (516h7m16.50810393s) [noise=true , address=145.79.227.22:4511 , piece_id=EWSHBP4MB2P6XTQFUYFHT5MVMYN74SUZSHZKBVEQ6YYFKQKZO3ZQ, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.637267104 (516h7m16.508350149s) [noise=true , address=77.90.31.10:4517 , piece_id=4CUKEE7ZXHZS3BKCGUBVTV5NGX2M6QIPX7UZ5AA643XDXLKQXB6Q, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.637545724 (516h7m16.508628759s) [noise=true , address=145.79.227.28:4727 , piece_id=IPW7CJRWRX3LOQVU72WYCH7N5A3AWCG7SOBPV3HLFUBLFKHWDD2Q, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.637951777 (516h7m16.509034822s) [noise=true , address=45.91.116.6:4973 , piece_id=Z6WHOPQKENWDHRJ6DNASRJYUU26YTB7ZAMIRW7SO2QRMT72RDZ7Q, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.638586496 (516h7m16.509669541s) [noise=true , address=145.79.227.51:4409 , piece_id=UKJH2XSSTG5OW7U2BUQIPIHS5MJ4X5FRHQOZO355AFFMNH3IGCTA, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.638981889 (516h7m16.510064934s) [noise=true , address=145.79.227.33:4869 , piece_id=WL44AIEAIVEOCAAROJ7G3XB7JBFOCBAZRDJECANSKKKSFE4LDSTQ, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.639704695 (516h7m16.510787731s) [noise=true , address=145.79.227.2:4114 , piece_id=IIFLOEDNW7RAADLO5RAGQKMIVASZ5563VTKBK2DV4QAD3DGHDBQA, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.640022180 (516h7m16.511105225s) [noise=true , address=145.79.227.13:4803 , piece_id=7DGXCXTR7ZOLSG3ZYQGHJH2VCYETKG4TCUC7B7SY4X2MHWTII2FA, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:16.640392324 (516h7m16.511475359s) [noise=true , address=145.79.227.38:4979 , piece_id=PO7KCJQNRHFOQCW3NJFKDRIWZ7J4QTSLMPOJQ3LAIYLIN5A5GPTA, seg_pos={0 0}, upload=452863]: Uploading piece... Dec 27 03:36:18.261737977 (516h7m18.132821022s) [upload=452863]: Next returned nil segment Dec 27 03:36:18.261752634 (516h7m18.13283567s) [upload=452863]: Waiting for error group managing segments... Dec 27 03:36:18.652422865 (516h7m18.52350591s) [noise=true , address=145.79.227.13:4803 , piece_id=7DGXCXTR7ZOLSG3ZYQGHJH2VCYETKG4TCUC7B7SY4X2MHWTII2FA, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:18.656046406 (516h7m18.527129441s) [noise=true , address=145.79.227.2:4114 , piece_id=IIFLOEDNW7RAADLO5RAGQKMIVASZ5563VTKBK2DV4QAD3DGHDBQA, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:18.656758552 (516h7m18.527841587s) [noise=true , address=145.79.227.15:4875 , piece_id=G22UQYJNIVIBG23GGN6LHF5JRWU5PRRMAWXWQN4VXICXMMDDAFCA, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:18.659088228 (516h7m18.530171273s) [noise=true , address=145.79.227.22:4511 , piece_id=EWSHBP4MB2P6XTQFUYFHT5MVMYN74SUZSHZKBVEQ6YYFKQKZO3ZQ, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:18.667446293 (516h7m18.538529328s) [noise=true , address=145.79.227.51:4409 , piece_id=UKJH2XSSTG5OW7U2BUQIPIHS5MJ4X5FRHQOZO355AFFMNH3IGCTA, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:18.668156596 (516h7m18.539239641s) [noise=true , address=145.79.227.28:4727 , piece_id=IPW7CJRWRX3LOQVU72WYCH7N5A3AWCG7SOBPV3HLFUBLFKHWDD2Q, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:18.671508829 (516h7m18.542591864s) [noise=true , address=145.79.227.38:4979 , piece_id=PO7KCJQNRHFOQCW3NJFKDRIWZ7J4QTSLMPOJQ3LAIYLIN5A5GPTA, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:18.836858176 (516h7m18.707941211s) [noise=true , address=145.79.227.33:4869 , piece_id=WL44AIEAIVEOCAAROJ7G3XB7JBFOCBAZRDJECANSKKKSFE4LDSTQ, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:18.871812015 (516h7m18.74289505s) [noise=true , address=77.90.31.10:4517 , piece_id=4CUKEE7ZXHZS3BKCGUBVTV5NGX2M6QIPX7UZ5AA643XDXLKQXB6Q, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:18.876291926 (516h7m18.747374961s) [noise=true , address=45.91.116.6:4973 , piece_id=Z6WHOPQKENWDHRJ6DNASRJYUU26YTB7ZAMIRW7SO2QRMT72RDZ7Q, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:18.999970473 (516h7m18.871053518s) [noise=true , address=45.91.116.13:4154 , piece_id=RNCIZFP3HEWLBIKH44W4MVP4SXDTUPESWSGKI22IEJLEFCC7MV7A, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:19.048131191 (516h7m18.919214236s) [noise=true , address=45.91.116.12:4226 , piece_id=2Z2DXYJZKOQAHJY4JLBDOYUKZ6LPTGZ6M2BHQMGVPK7RESMP36XA, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:19.249019370 (516h7m19.120102405s) [noise=true , address=77.90.31.14:4661 , piece_id=ZXOOFBTBKHRXHX5M2OULO44ZIHOJJ2UBKUWOIVY3MNXG7PB7Y34Q, seg_pos={0 0}, upload=452863]: Done uploading piece. err: <nil> Dec 27 03:36:19.249039208 (516h7m19.120122253s) [seg_pos={0 0}, upload=452863]: Segment reached optimal threshold of 13 pieces. Dec 27 03:36:19.249212368 (516h7m19.120295403s) [noise=true , address=145.79.227.34:4907 , piece_id=ASSTOYCTSFKC3MDPIAQIG2ON6AGQKHWZ7O2GM76ISKBJERPOX7VQ, seg_pos={0 0}, upload=452863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 03:36:19.249220794 (516h7m19.120303829s) [noise=true , address=145.79.227.34:4907 , piece_id=ASSTOYCTSFKC3MDPIAQIG2ON6AGQKHWZ7O2GM76ISKBJERPOX7VQ, seg_pos={0 0}, upload=452863]: Long tail cancellation due to optimal threshold Dec 27 03:36:19.249346083 (516h7m19.120429118s) [noise=true , address=77.90.31.12:4590 , piece_id=Q4PN6ZOUE6F7P3NR67DQYXHU7VF2LPZML2HFMNUMSGC4LBYCV3YA, seg_pos={0 0}, upload=452863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 03:36:19.249365419 (516h7m19.120448454s) [noise=true , address=77.90.31.12:4590 , piece_id=Q4PN6ZOUE6F7P3NR67DQYXHU7VF2LPZML2HFMNUMSGC4LBYCV3YA, seg_pos={0 0}, upload=452863]: Long tail cancellation due to optimal threshold Dec 27 03:36:19.249389775 (516h7m19.120472811s) [noise=true , address=45.91.116.9:4262 , piece_id=IKBNJQWTAHCEIAU4EFDYMXVC3K7I7K2SACKULT2D6SCG6S7AWMEA, seg_pos={0 0}, upload=452863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:42706->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:42706->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42706->45.91.116.9:4262: use of closed network connection Dec 27 03:36:19.249427487 (516h7m19.120510552s) [noise=true , address=45.91.116.9:4262 , piece_id=IKBNJQWTAHCEIAU4EFDYMXVC3K7I7K2SACKULT2D6SCG6S7AWMEA, seg_pos={0 0}, upload=452863]: Long tail cancellation due to optimal threshold Dec 27 03:36:19.249471501 (516h7m19.120554536s) [seg_pos={0 0}, upload=452863]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:36:19.249629181 (516h7m19.120712217s) [upload=452863]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:36:19.249636034 (516h7m19.120719069s) [upload=452863]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:36:19.549946443 (516h7m19.421029478s) [upload=452863]: Done uploading segments... Dec 27 03:36:19.549969206 (516h7m19.421052241s) [upload=452863]: Upload finished. err: <nil> Duration: 30.002842856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [.X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.XX] duration: 30.004274101s trace: https://jaeger.dev.storj.io/trace/520a25f4a66fc5cb FAIL (2025-12-27 03:36:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 6D46JQJR4C36QSE5OO3JXPG5MOVT2UB4UBFTEJQZHWDK4PQUEDRA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: RZFMLYIO65TBPWYUQKSLDYKMOC2L6EURLCWPO3XJIDJ3UUE6SBHA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: TN6IFFCG4B6ORJOZQKL6SJBMRAPCN4XBGKLXCQ2V6PMWWGWTKZWQ) context deadline exceeded ======LOGS====== em: pb.BatchRequestItem_ObjectBegin Dec 27 03:35:32.171865902 (516h6m32.042948938s) [upload=452853]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:35:32.420811852 (516h6m32.291894887s) [seg_pos={0 0}, upload=452853]: Begin upload segment... Dec 27 03:35:32.420855495 (516h6m32.29193853s) [seg_pos={0 0}, upload=452853]: Done begin upload segment. Dec 27 03:35:32.420882116 (516h6m32.291965151s) [noise=true , address=145.79.227.51:4410 , piece_id=5DJWLO2KK57ICIKC2E4DE7ULGSEVJI2ZNV6KTXSMXJG2AG7P7KMA, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.420956768 (516h6m32.292039813s) [noise=true , address=45.91.116.9:4261 , piece_id=CSZUX4QLHYXXZUHSHLXLCFY4EF2LCLG3Z4UWNXGWK6VFYHVTLFCA, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.421469484 (516h6m32.29255252s) [noise=true , address=45.91.116.8:4048 , piece_id=RZFMLYIO65TBPWYUQKSLDYKMOC2L6EURLCWPO3XJIDJ3UUE6SBHA, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.421856111 (516h6m32.292939166s) [noise=true , address=45.91.116.10:4084 , piece_id=TN6IFFCG4B6ORJOZQKL6SJBMRAPCN4XBGKLXCQ2V6PMWWGWTKZWQ, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.422220936 (516h6m32.293303971s) [noise=true , address=145.79.227.47:4265 , piece_id=3WV3NWUTJK6PFS44V2SQDHYDENJUGQBPFFOGPV7WCT26SYFEXSJA, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.422634133 (516h6m32.293717168s) [noise=true , address=145.79.227.2:4115 , piece_id=JKYDFNTDFSN5UDQQJT76KTHWN5M6QH2BZJOKHLZ7SX7VGCJBLKTQ, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.422975442 (516h6m32.294058477s) [noise=true , address=145.79.227.48:4302 , piece_id=XDCNU2LBP37D3SCHHYWGRDRVAJO5NSBUPN3M72W66457H7L6EY4A, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.423306563 (516h6m32.294389598s) [noise=true , address=45.91.116.14:4191 , piece_id=GCEWW3XSQJW4NCES4GR7S2XLAWUBE7TQVVD7BJI5ZAGOCDCU7W2Q, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.423658423 (516h6m32.294741458s) [noise=true , address=145.79.227.42:4122 , piece_id=KHSSYGLTDMIXFNJNTT5VRE3LWTWMKLDEJT3YWJJB3OSQMOHJYOMA, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.424000524 (516h6m32.295083559s) [noise=true , address=77.90.31.11:4553 , piece_id=XEN7MSRYTMCSV3KBD26Z7DWOSWRK65NJ4QVNPNB5EHYCAELYTLRA, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.426032674 (516h6m32.297115709s) [noise=true , address=45.91.116.13:4154 , piece_id=6D46JQJR4C36QSE5OO3JXPG5MOVT2UB4UBFTEJQZHWDK4PQUEDRA, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.426537405 (516h6m32.29762045s) [noise=true , address=145.79.227.15:4873 , piece_id=QSHPLOGTGUGMF5DA74GY35EVPFFYKPAFKMDKSLTMTNBQ2O7L3BQA, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.426903391 (516h6m32.297986427s) [noise=true , address=145.79.227.49:4337 , piece_id=H4ORGDCRYO4FKRG5ZBRBFDBTK5FWVPUGNSGVXIGLHJ2452Z4HWWQ, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.427328672 (516h6m32.298411707s) [noise=true , address=145.79.227.16:4912 , piece_id=JHJ22GVIGPECKBOAVBAYFWTCJDEWYGCYUNHJBND2JNMHKX5TJOPQ, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.427741879 (516h6m32.298824924s) [noise=true , address=45.91.116.4:4939 , piece_id=ZQBNC65FOINBRISPAAILDUUNIZP5DONOESFBVBLVYWE4YBLFFVJA, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:32.429256643 (516h6m32.300339688s) [noise=true , address=145.79.227.5:4222 , piece_id=CM4EIBIJFJ3WE2IPCU3AKDAAOQL55ZJC7OVYGWD5LLF6WKUC3KIA, seg_pos={0 0}, upload=452853]: Uploading piece... Dec 27 03:35:34.138782127 (516h6m34.009865162s) [upload=452853]: Next returned nil segment Dec 27 03:35:34.138795993 (516h6m34.009879028s) [upload=452853]: Waiting for error group managing segments... Dec 27 03:35:34.423930865 (516h6m34.2950139s) [noise=true , address=145.79.227.48:4302 , piece_id=XDCNU2LBP37D3SCHHYWGRDRVAJO5NSBUPN3M72W66457H7L6EY4A, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.427505131 (516h6m34.298588166s) [noise=true , address=145.79.227.49:4337 , piece_id=H4ORGDCRYO4FKRG5ZBRBFDBTK5FWVPUGNSGVXIGLHJ2452Z4HWWQ, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.427775666 (516h6m34.298858701s) [noise=true , address=145.79.227.15:4873 , piece_id=QSHPLOGTGUGMF5DA74GY35EVPFFYKPAFKMDKSLTMTNBQ2O7L3BQA, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.450021878 (516h6m34.321104913s) [noise=true , address=145.79.227.16:4912 , piece_id=JHJ22GVIGPECKBOAVBAYFWTCJDEWYGCYUNHJBND2JNMHKX5TJOPQ, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.453131438 (516h6m34.324214484s) [noise=true , address=145.79.227.2:4115 , piece_id=JKYDFNTDFSN5UDQQJT76KTHWN5M6QH2BZJOKHLZ7SX7VGCJBLKTQ, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.454951905 (516h6m34.32603495s) [noise=true , address=145.79.227.51:4410 , piece_id=5DJWLO2KK57ICIKC2E4DE7ULGSEVJI2ZNV6KTXSMXJG2AG7P7KMA, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.455560183 (516h6m34.326643259s) [noise=true , address=145.79.227.42:4122 , piece_id=KHSSYGLTDMIXFNJNTT5VRE3LWTWMKLDEJT3YWJJB3OSQMOHJYOMA, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.459837037 (516h6m34.330920082s) [noise=true , address=145.79.227.47:4265 , piece_id=3WV3NWUTJK6PFS44V2SQDHYDENJUGQBPFFOGPV7WCT26SYFEXSJA, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.461736123 (516h6m34.332819158s) [noise=true , address=145.79.227.5:4222 , piece_id=CM4EIBIJFJ3WE2IPCU3AKDAAOQL55ZJC7OVYGWD5LLF6WKUC3KIA, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.647588242 (516h6m34.518671277s) [noise=true , address=45.91.116.13:4154 , piece_id=6D46JQJR4C36QSE5OO3JXPG5MOVT2UB4UBFTEJQZHWDK4PQUEDRA, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.702318007 (516h6m34.573401042s) [noise=true , address=45.91.116.10:4084 , piece_id=TN6IFFCG4B6ORJOZQKL6SJBMRAPCN4XBGKLXCQ2V6PMWWGWTKZWQ, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:34.987879401 (516h6m34.858962436s) [noise=true , address=45.91.116.14:4191 , piece_id=GCEWW3XSQJW4NCES4GR7S2XLAWUBE7TQVVD7BJI5ZAGOCDCU7W2Q, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:35.025697224 (516h6m34.896780259s) [noise=true , address=45.91.116.8:4048 , piece_id=RZFMLYIO65TBPWYUQKSLDYKMOC2L6EURLCWPO3XJIDJ3UUE6SBHA, seg_pos={0 0}, upload=452853]: Done uploading piece. err: <nil> Dec 27 03:35:35.025716180 (516h6m34.896799215s) [seg_pos={0 0}, upload=452853]: Segment reached optimal threshold of 13 pieces. Dec 27 03:35:35.025923685 (516h6m34.89700672s) [noise=true , address=45.91.116.9:4261 , piece_id=CSZUX4QLHYXXZUHSHLXLCFY4EF2LCLG3Z4UWNXGWK6VFYHVTLFCA, seg_pos={0 0}, upload=452853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:46602->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:46602->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46602->45.91.116.9:4261: use of closed network connection Dec 27 03:35:35.025929546 (516h6m34.897012581s) [noise=true , address=45.91.116.9:4261 , piece_id=CSZUX4QLHYXXZUHSHLXLCFY4EF2LCLG3Z4UWNXGWK6VFYHVTLFCA, seg_pos={0 0}, upload=452853]: Long tail cancellation due to optimal threshold Dec 27 03:35:35.026047370 (516h6m34.897130405s) [noise=true , address=77.90.31.11:4553 , piece_id=XEN7MSRYTMCSV3KBD26Z7DWOSWRK65NJ4QVNPNB5EHYCAELYTLRA, seg_pos={0 0}, upload=452853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43374->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43374->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43374->77.90.31.11:4553: use of closed network connection Dec 27 03:35:35.026053371 (516h6m34.897136407s) [noise=true , address=77.90.31.11:4553 , piece_id=XEN7MSRYTMCSV3KBD26Z7DWOSWRK65NJ4QVNPNB5EHYCAELYTLRA, seg_pos={0 0}, upload=452853]: Long tail cancellation due to optimal threshold Dec 27 03:35:35.026743967 (516h6m34.897827022s) [noise=true , address=45.91.116.4:4939 , piece_id=ZQBNC65FOINBRISPAAILDUUNIZP5DONOESFBVBLVYWE4YBLFFVJA, seg_pos={0 0}, upload=452853]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:38448->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:38448->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->45.91.116.4:4939: use of closed network connection Dec 27 03:35:35.026771199 (516h6m34.897854254s) [noise=true , address=45.91.116.4:4939 , piece_id=ZQBNC65FOINBRISPAAILDUUNIZP5DONOESFBVBLVYWE4YBLFFVJA, seg_pos={0 0}, upload=452853]: Long tail cancellation due to optimal threshold Dec 27 03:35:35.026824570 (516h6m34.897907605s) [seg_pos={0 0}, upload=452853]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:35:35.026937736 (516h6m34.898020791s) [upload=452853]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:35:35.026944659 (516h6m34.898027704s) [upload=452853]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:35:35.320568715 (516h6m35.19165176s) [upload=452853]: Done uploading segments... Dec 27 03:35:35.320591869 (516h6m35.191674905s) [upload=452853]: Upload finished. err: <nil> Duration: 30.004302236s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [..X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.X] duration: 30.002837362s trace: https://jaeger.dev.storj.io/trace/60e9c392dc4b1155 FAIL (2025-12-27 03:35:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: II6BSCAMZWHTKX72T5YMZB2FU7NG2QG5OT6Z33LA46BNRTPCJJEQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KFSVSHNTPTOL5ZTPIID7ORDAO32LLITQBW2GCX4V3LAWWZSPILJA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: QRCXPGGLNV45XBHZP3OYTCZPUJ3Z5Z6RIMWE22IGRIKLYSDGHXVQ) context deadline exceeded ======LOGS====== Dec 27 03:35:01.147385425 (516h6m1.01846846s) [upload=452844]: Starting upload Dec 27 03:35:01.147449857 (516h6m1.018532902s) [upload=452844]: Done starting upload Dec 27 03:35:01.147677781 (516h6m1.018760826s) [upload=452844]: Uploading segments... Dec 27 03:35:01.147744228 (516h6m1.018827293s) [upload=452844]: Got next segment. Inline: false Dec 27 03:35:01.147785376 (516h6m1.018868421s) [upload=452844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:35:01.147795705 (516h6m1.01887878s) [upload=452844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:35:01.393217846 (516h6m1.264300901s) [seg_pos={0 0}, upload=452844]: Begin upload segment... Dec 27 03:35:01.393294162 (516h6m1.264377207s) [seg_pos={0 0}, upload=452844]: Done begin upload segment. Dec 27 03:35:01.393337635 (516h6m1.26442067s) [noise=true , address=45.91.116.14:4190 , piece_id=MAOZSN3WUNHYTH3GI6XCIY4SLGCAB4J5XCMVHFS4ACUMDJC5T6NQ, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.393363364 (516h6m1.264446399s) [noise=true , address=45.91.116.7:4009 , piece_id=C5CQYM6HZOMGV2ZN6BQ37ZBN74HRAFE5FNOIGTLHQTSIZAP25Q5Q, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.394192753 (516h6m1.265275798s) [noise=true , address=45.91.116.8:4046 , piece_id=3OF62UDLD5S3VI7MG5XO6DZAZR3IVBGMG75I6K3OHWTDKYBTJF2Q, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.394710629 (516h6m1.265793664s) [noise=true , address=77.90.31.15:4697 , piece_id=KFSVSHNTPTOL5ZTPIID7ORDAO32LLITQBW2GCX4V3LAWWZSPILJA, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.395033964 (516h6m1.266117009s) [noise=true , address=145.79.227.48:4302 , piece_id=R7YDFS4GPXI34KMVM6HG2HM2FJ7GV2ITVKF2CQZTS3U2LTXOMINA, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.395361789 (516h6m1.266444834s) [noise=true , address=145.79.227.24:4582 , piece_id=KHS2PLOIVB546SMFNSE25MJP24REO66VHB4M5TYWB5DELCQ7NJWA, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.395721975 (516h6m1.26680502s) [noise=true , address=145.79.227.47:4267 , piece_id=4N4VWUWCDD4H3HIZE5VPESDJX5XPAQTVVPOA6WYCWX7MU6YGP2VA, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.397283508 (516h6m1.268366553s) [noise=true , address=145.79.227.22:4511 , piece_id=2A25NJCZL67EG5TFFTKRXEWGA4G5IGGXMXWAT4HF2ZBWULNYO7LA, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.398631525 (516h6m1.26971457s) [noise=true , address=145.79.227.28:4725 , piece_id=L2PYHCHQR6CBONKXPUBBAVWR4U2HWPLSCET5JU4TSVTSPFLQZRGQ, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.400629560 (516h6m1.271712605s) [noise=true , address=45.91.116.6:4973 , piece_id=JZ34KI34NSWWC2FHOURJGYLYEBO25PY4DWAOJV5SMIGRTJ7SFMXQ, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.402099398 (516h6m1.273182443s) [noise=true , address=145.79.227.11:4729 , piece_id=ZY6523AUUCMC2A4FLL6AF7UFBHVRRFCH3LPCEB5JQCCS2SEYLHRQ, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.403441985 (516h6m1.27452503s) [noise=true , address=45.91.116.5:4298 , piece_id=QRCXPGGLNV45XBHZP3OYTCZPUJ3Z5Z6RIMWE22IGRIKLYSDGHXVQ, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.406368217 (516h6m1.277451252s) [noise=true , address=145.79.227.50:4374 , piece_id=V73IIUXHBYPPAP3OSXGWU7IN5A55AGKCAWZWUSLN6PJJ3KUIYVJQ, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.410560470 (516h6m1.281643505s) [noise=true , address=145.79.227.44:4194 , piece_id=TRM4XQQ5VQUPQG755W4CWHOLFHNA3UGFVRKGWXPHHGFATLJIRQEA, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.414692077 (516h6m1.285775443s) [noise=true , address=45.91.116.13:4154 , piece_id=II6BSCAMZWHTKX72T5YMZB2FU7NG2QG5OT6Z33LA46BNRTPCJJEQ, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:01.415055690 (516h6m1.286138735s) [noise=true , address=45.91.116.9:4263 , piece_id=HBMTW5XQ7KYBAGZZXDVORAS3GA2N73WG4EJHHR5D7OTA7SFITBCQ, seg_pos={0 0}, upload=452844]: Uploading piece... Dec 27 03:35:03.144043745 (516h6m3.01512679s) [upload=452844]: Next returned nil segment Dec 27 03:35:03.144064695 (516h6m3.01514774s) [upload=452844]: Waiting for error group managing segments... Dec 27 03:35:03.424397944 (516h6m3.295480989s) [noise=true , address=145.79.227.48:4302 , piece_id=R7YDFS4GPXI34KMVM6HG2HM2FJ7GV2ITVKF2CQZTS3U2LTXOMINA, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.455674710 (516h6m3.326757745s) [noise=true , address=145.79.227.22:4511 , piece_id=2A25NJCZL67EG5TFFTKRXEWGA4G5IGGXMXWAT4HF2ZBWULNYO7LA, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.458642882 (516h6m3.329725927s) [noise=true , address=145.79.227.28:4725 , piece_id=L2PYHCHQR6CBONKXPUBBAVWR4U2HWPLSCET5JU4TSVTSPFLQZRGQ, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.462052364 (516h6m3.333135409s) [noise=true , address=145.79.227.11:4729 , piece_id=ZY6523AUUCMC2A4FLL6AF7UFBHVRRFCH3LPCEB5JQCCS2SEYLHRQ, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.465078447 (516h6m3.336161502s) [noise=true , address=145.79.227.50:4374 , piece_id=V73IIUXHBYPPAP3OSXGWU7IN5A55AGKCAWZWUSLN6PJJ3KUIYVJQ, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.479852490 (516h6m3.350935525s) [noise=true , address=145.79.227.47:4267 , piece_id=4N4VWUWCDD4H3HIZE5VPESDJX5XPAQTVVPOA6WYCWX7MU6YGP2VA, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.493520927 (516h6m3.364603972s) [noise=true , address=145.79.227.24:4582 , piece_id=KHS2PLOIVB546SMFNSE25MJP24REO66VHB4M5TYWB5DELCQ7NJWA, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.710872436 (516h6m3.581955481s) [noise=true , address=77.90.31.15:4697 , piece_id=KFSVSHNTPTOL5ZTPIID7ORDAO32LLITQBW2GCX4V3LAWWZSPILJA, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.739419745 (516h6m3.61050279s) [noise=true , address=45.91.116.8:4046 , piece_id=3OF62UDLD5S3VI7MG5XO6DZAZR3IVBGMG75I6K3OHWTDKYBTJF2Q, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.754739267 (516h6m3.625822322s) [noise=true , address=45.91.116.6:4973 , piece_id=JZ34KI34NSWWC2FHOURJGYLYEBO25PY4DWAOJV5SMIGRTJ7SFMXQ, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.759604852 (516h6m3.630687897s) [noise=true , address=45.91.116.13:4154 , piece_id=II6BSCAMZWHTKX72T5YMZB2FU7NG2QG5OT6Z33LA46BNRTPCJJEQ, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.775843252 (516h6m3.646926297s) [noise=true , address=45.91.116.14:4190 , piece_id=MAOZSN3WUNHYTH3GI6XCIY4SLGCAB4J5XCMVHFS4ACUMDJC5T6NQ, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.828155926 (516h6m3.699238971s) [noise=true , address=45.91.116.5:4298 , piece_id=QRCXPGGLNV45XBHZP3OYTCZPUJ3Z5Z6RIMWE22IGRIKLYSDGHXVQ, seg_pos={0 0}, upload=452844]: Done uploading piece. err: <nil> Dec 27 03:35:03.828185272 (516h6m3.699268307s) [seg_pos={0 0}, upload=452844]: Segment reached optimal threshold of 13 pieces. Dec 27 03:35:03.828450898 (516h6m3.699533953s) [noise=true , address=45.91.116.7:4009 , piece_id=C5CQYM6HZOMGV2ZN6BQ37ZBN74HRAFE5FNOIGTLHQTSIZAP25Q5Q, seg_pos={0 0}, upload=452844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 03:35:03.828465626 (516h6m3.699548671s) [noise=true , address=45.91.116.7:4009 , piece_id=C5CQYM6HZOMGV2ZN6BQ37ZBN74HRAFE5FNOIGTLHQTSIZAP25Q5Q, seg_pos={0 0}, upload=452844]: Long tail cancellation due to optimal threshold Dec 27 03:35:03.828577218 (516h6m3.699660273s) [noise=true , address=145.79.227.44:4194 , piece_id=TRM4XQQ5VQUPQG755W4CWHOLFHNA3UGFVRKGWXPHHGFATLJIRQEA, seg_pos={0 0}, upload=452844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 03:35:03.828586396 (516h6m3.699669481s) [noise=true , address=145.79.227.44:4194 , piece_id=TRM4XQQ5VQUPQG755W4CWHOLFHNA3UGFVRKGWXPHHGFATLJIRQEA, seg_pos={0 0}, upload=452844]: Long tail cancellation due to optimal threshold Dec 27 03:35:03.828671959 (516h6m3.699755004s) [noise=true , address=45.91.116.9:4263 , piece_id=HBMTW5XQ7KYBAGZZXDVORAS3GA2N73WG4EJHHR5D7OTA7SFITBCQ, seg_pos={0 0}, upload=452844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: read tcp 188.166.225.170:37998->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:35:03.828699191 (516h6m3.699782236s) [noise=true , address=45.91.116.9:4263 , piece_id=HBMTW5XQ7KYBAGZZXDVORAS3GA2N73WG4EJHHR5D7OTA7SFITBCQ, seg_pos={0 0}, upload=452844]: Long tail cancellation due to optimal threshold Dec 27 03:35:03.828722585 (516h6m3.699805631s) [seg_pos={0 0}, upload=452844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:35:03.828881618 (516h6m3.699964673s) [upload=452844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:35:03.828890194 (516h6m3.699973239s) [upload=452844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:35:04.117044215 (516h6m3.98812727s) [upload=452844]: Done uploading segments... Dec 27 03:35:04.117073541 (516h6m3.988156586s) [upload=452844]: Upload finished. err: <nil> Duration: 30.002861498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX.] duration: 6.826909212s trace: https://jaeger.dev.storj.io/trace/4ecc64f58880504b FAIL (2025-12-27 03:34:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: HWHEVCN7OU6N343CCUYQ463OVJX7VE2WY4JWGPST4IVSAD746SDA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 46GFR7YBSFWWZ4FKKH7NRRIKWHPUKM2SV5CPVJQVYJV4YOOCYXEQ) context deadline exceeded ======LOGS====== Dec 27 03:34:04.240193938 (516h5m4.111276983s) [upload=452830]: Starting upload Dec 27 03:34:04.240269612 (516h5m4.111352647s) [upload=452830]: Done starting upload Dec 27 03:34:04.240328394 (516h5m4.111411439s) [upload=452830]: Uploading segments... Dec 27 03:34:04.240570225 (516h5m4.11165328s) [upload=452830]: Got next segment. Inline: false Dec 27 03:34:04.240621031 (516h5m4.111704066s) [upload=452830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:34:04.240626762 (516h5m4.111709797s) [upload=452830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:34:04.485660172 (516h5m4.356743228s) [seg_pos={0 0}, upload=452830]: Begin upload segment... Dec 27 03:34:04.485760904 (516h5m4.356843949s) [seg_pos={0 0}, upload=452830]: Done begin upload segment. Dec 27 03:34:04.485853510 (516h5m4.356936555s) [noise=true , address=145.79.227.36:4519 , piece_id=WEAKNDM3ZCPZ6JOK6FTKKLV2GUSDGKPL24V5AY4XHXN6YXLOCB3A, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.485921299 (516h5m4.357004344s) [noise=true , address=145.79.227.37:4566 , piece_id=BIIGCSDWIXW7M462LUQCZSOIWPC4DZDCDCLDWXJOPI445LZPD26A, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.486834619 (516h5m4.357917654s) [noise=true , address=77.90.31.13:4627 , piece_id=HWHEVCN7OU6N343CCUYQ463OVJX7VE2WY4JWGPST4IVSAD746SDA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.487234680 (516h5m4.358317725s) [noise=true , address=77.90.31.10:4520 , piece_id=43XIRJESKJMQFRWCZ5RWCE6K4RLJWOXSA7LSHSEZNSFQ7XUPSC4A, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.487715486 (516h5m4.358798531s) [noise=true , address=45.91.116.12:4225 , piece_id=VPYHHXTTZOX5WFCY2YDC5PAF4A5HWKA7TJY2BCOMQ6MLR47LULBA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.488121640 (516h5m4.359204675s) [noise=true , address=145.79.227.52:4446 , piece_id=KKPOCFIESLU35ZBHJCY5O25JBWUV6IZAUYIMJ6QYHKANFRI7DITQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.488516571 (516h5m4.359599617s) [noise=true , address=145.79.227.45:4231 , piece_id=7AVKMKZAXMD7KNYJFV6BN6KT5NSWXSIWJYDLIY3AHMWAGVNMVQIQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.489056901 (516h5m4.360139946s) [noise=true , address=77.90.31.11:4554 , piece_id=46GFR7YBSFWWZ4FKKH7NRRIKWHPUKM2SV5CPVJQVYJV4YOOCYXEQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.490160132 (516h5m4.361243177s) [noise=true , address=145.79.227.10:4401 , piece_id=JMKNAW2HKBMMVNXR3R4R36OY4MKPKYLYJ2DD7ZMYH7VSQODIRHTA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.490724516 (516h5m4.361807561s) [noise=true , address=145.79.227.49:4339 , piece_id=D5V5NWUBUFL2CHUPRITXISTUABMN4IKWBPKXDQMZN2U3MFL3SGFQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.491214810 (516h5m4.362297855s) [noise=true , address=145.79.227.11:4731 , piece_id=7XK2P2JTDR7UTDQODNJ3JSC6Q66NPJAIUYLDXSM3M5HZOSYY4FGA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.491977733 (516h5m4.363060768s) [noise=true , address=145.79.227.9:4365 , piece_id=6X5CNCUDD3VZJWBEUP7IB76XWO6WXXTY6I4JE7WAEYSG6U7E2LKQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.492564941 (516h5m4.363647986s) [noise=true , address=45.91.116.7:4010 , piece_id=57WNHJAQWGNICJ5GU2I6MTVWOFH2C6A67LF3SWA47U56MBBWVBZA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.493203927 (516h5m4.364286972s) [noise=true , address=77.90.31.15:4700 , piece_id=D4J2A44AKEVYYAT3UT5Y57DOKCEE4ISTO27EDF4FDB2KRWSSA4WA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.494295506 (516h5m4.365378551s) [noise=true , address=145.79.227.43:4158 , piece_id=L2OG5DUNLCLYQOAPCURAEJ2N2HXHKXLTDYZSPZ2KYB5JOSSBUWAA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.494913162 (516h5m4.365996207s) [noise=true , address=45.91.116.10:4081 , piece_id=ZB4XT6LXC7GQQOJQ5IPEMWAHE7LCJV3ISQEU455ZCEYANZUU3EXQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:06.107398013 (516h5m5.978481048s) [upload=452830]: Next returned nil segment Dec 27 03:34:06.107411869 (516h5m5.978494904s) [upload=452830]: Waiting for error group managing segments... Dec 27 03:34:06.495613799 (516h5m6.366696834s) [noise=true , address=145.79.227.10:4401 , piece_id=JMKNAW2HKBMMVNXR3R4R36OY4MKPKYLYJ2DD7ZMYH7VSQODIRHTA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.496075158 (516h5m6.367158203s) [noise=true , address=145.79.227.9:4365 , piece_id=6X5CNCUDD3VZJWBEUP7IB76XWO6WXXTY6I4JE7WAEYSG6U7E2LKQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.501918784 (516h5m6.373001829s) [noise=true , address=145.79.227.45:4231 , piece_id=7AVKMKZAXMD7KNYJFV6BN6KT5NSWXSIWJYDLIY3AHMWAGVNMVQIQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.502537422 (516h5m6.373620468s) [noise=true , address=145.79.227.37:4566 , piece_id=BIIGCSDWIXW7M462LUQCZSOIWPC4DZDCDCLDWXJOPI445LZPD26A, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.510486179 (516h5m6.381569224s) [noise=true , address=145.79.227.52:4446 , piece_id=KKPOCFIESLU35ZBHJCY5O25JBWUV6IZAUYIMJ6QYHKANFRI7DITQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.518088785 (516h5m6.38917182s) [noise=true , address=145.79.227.11:4731 , piece_id=7XK2P2JTDR7UTDQODNJ3JSC6Q66NPJAIUYLDXSM3M5HZOSYY4FGA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.520758940 (516h5m6.391841985s) [noise=true , address=145.79.227.36:4519 , piece_id=WEAKNDM3ZCPZ6JOK6FTKKLV2GUSDGKPL24V5AY4XHXN6YXLOCB3A, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.530331888 (516h5m6.401414934s) [noise=true , address=145.79.227.49:4339 , piece_id=D5V5NWUBUFL2CHUPRITXISTUABMN4IKWBPKXDQMZN2U3MFL3SGFQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.679953787 (516h5m6.551036832s) [noise=true , address=77.90.31.13:4627 , piece_id=HWHEVCN7OU6N343CCUYQ463OVJX7VE2WY4JWGPST4IVSAD746SDA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.777916247 (516h5m6.648999282s) [noise=true , address=45.91.116.7:4010 , piece_id=57WNHJAQWGNICJ5GU2I6MTVWOFH2C6A67LF3SWA47U56MBBWVBZA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.797929275 (516h5m6.66901231s) [noise=true , address=77.90.31.11:4554 , piece_id=46GFR7YBSFWWZ4FKKH7NRRIKWHPUKM2SV5CPVJQVYJV4YOOCYXEQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.931457328 (516h5m6.802540373s) [noise=true , address=45.91.116.10:4081 , piece_id=ZB4XT6LXC7GQQOJQ5IPEMWAHE7LCJV3ISQEU455ZCEYANZUU3EXQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:07.024167666 (516h5m6.895250711s) [noise=true , address=45.91.116.12:4225 , piece_id=VPYHHXTTZOX5WFCY2YDC5PAF4A5HWKA7TJY2BCOMQ6MLR47LULBA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:07.024208094 (516h5m6.895291129s) [seg_pos={0 0}, upload=452830]: Segment reached optimal threshold of 13 pieces. Dec 27 03:34:07.024474891 (516h5m6.895557926s) [noise=true , address=77.90.31.15:4700 , piece_id=D4J2A44AKEVYYAT3UT5Y57DOKCEE4ISTO27EDF4FDB2KRWSSA4WA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:56044->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:56044->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56044->77.90.31.15:4700: use of closed network connection Dec 27 03:34:07.024491623 (516h5m6.895574658s) [noise=true , address=77.90.31.15:4700 , piece_id=D4J2A44AKEVYYAT3UT5Y57DOKCEE4ISTO27EDF4FDB2KRWSSA4WA, seg_pos={0 0}, upload=452830]: Long tail cancellation due to optimal threshold Dec 27 03:34:07.024565113 (516h5m6.895648158s) [noise=true , address=77.90.31.10:4520 , piece_id=43XIRJESKJMQFRWCZ5RWCE6K4RLJWOXSA7LSHSEZNSFQ7XUPSC4A, seg_pos={0 0}, upload=452830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 03:34:07.024573990 (516h5m6.895657025s) [noise=true , address=77.90.31.10:4520 , piece_id=43XIRJESKJMQFRWCZ5RWCE6K4RLJWOXSA7LSHSEZNSFQ7XUPSC4A, seg_pos={0 0}, upload=452830]: Long tail cancellation due to optimal threshold Dec 27 03:34:07.024755435 (516h5m6.89583848s) [noise=true , address=145.79.227.43:4158 , piece_id=L2OG5DUNLCLYQOAPCURAEJ2N2HXHKXLTDYZSPZ2KYB5JOSSBUWAA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 03:34:07.024768240 (516h5m6.895851285s) [noise=true , address=145.79.227.43:4158 , piece_id=L2OG5DUNLCLYQOAPCURAEJ2N2HXHKXLTDYZSPZ2KYB5JOSSBUWAA, seg_pos={0 0}, upload=452830]: Long tail cancellation due to optimal threshold Dec 27 03:34:07.024813075 (516h5m6.89589612s) [seg_pos={0 0}, upload=452830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:34:07.024909539 (516h5m6.895992574s) [upload=452830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:34:07.024914609 (516h5m6.895997644s) [upload=452830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:34:07.316939289 (516h5m7.188022324s) [upload=452830]: Done uploading segments... Dec 27 03:34:07.316964897 (516h5m7.188047942s) [upload=452830]: Upload finished. err: <nil> Duration: 6.82692374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [X...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXXX] duration: 30.002705422s trace: https://jaeger.dev.storj.io/trace/4ecc64f58880504b FAIL (2025-12-27 03:34:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: HWHEVCN7OU6N343CCUYQ463OVJX7VE2WY4JWGPST4IVSAD746SDA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 46GFR7YBSFWWZ4FKKH7NRRIKWHPUKM2SV5CPVJQVYJV4YOOCYXEQ) context deadline exceeded ======LOGS====== Dec 27 03:34:04.240193938 (516h5m4.111276983s) [upload=452830]: Starting upload Dec 27 03:34:04.240269612 (516h5m4.111352647s) [upload=452830]: Done starting upload Dec 27 03:34:04.240328394 (516h5m4.111411439s) [upload=452830]: Uploading segments... Dec 27 03:34:04.240570225 (516h5m4.11165328s) [upload=452830]: Got next segment. Inline: false Dec 27 03:34:04.240621031 (516h5m4.111704066s) [upload=452830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:34:04.240626762 (516h5m4.111709797s) [upload=452830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:34:04.485660172 (516h5m4.356743228s) [seg_pos={0 0}, upload=452830]: Begin upload segment... Dec 27 03:34:04.485760904 (516h5m4.356843949s) [seg_pos={0 0}, upload=452830]: Done begin upload segment. Dec 27 03:34:04.485853510 (516h5m4.356936555s) [noise=true , address=145.79.227.36:4519 , piece_id=WEAKNDM3ZCPZ6JOK6FTKKLV2GUSDGKPL24V5AY4XHXN6YXLOCB3A, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.485921299 (516h5m4.357004344s) [noise=true , address=145.79.227.37:4566 , piece_id=BIIGCSDWIXW7M462LUQCZSOIWPC4DZDCDCLDWXJOPI445LZPD26A, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.486834619 (516h5m4.357917654s) [noise=true , address=77.90.31.13:4627 , piece_id=HWHEVCN7OU6N343CCUYQ463OVJX7VE2WY4JWGPST4IVSAD746SDA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.487234680 (516h5m4.358317725s) [noise=true , address=77.90.31.10:4520 , piece_id=43XIRJESKJMQFRWCZ5RWCE6K4RLJWOXSA7LSHSEZNSFQ7XUPSC4A, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.487715486 (516h5m4.358798531s) [noise=true , address=45.91.116.12:4225 , piece_id=VPYHHXTTZOX5WFCY2YDC5PAF4A5HWKA7TJY2BCOMQ6MLR47LULBA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.488121640 (516h5m4.359204675s) [noise=true , address=145.79.227.52:4446 , piece_id=KKPOCFIESLU35ZBHJCY5O25JBWUV6IZAUYIMJ6QYHKANFRI7DITQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.488516571 (516h5m4.359599617s) [noise=true , address=145.79.227.45:4231 , piece_id=7AVKMKZAXMD7KNYJFV6BN6KT5NSWXSIWJYDLIY3AHMWAGVNMVQIQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.489056901 (516h5m4.360139946s) [noise=true , address=77.90.31.11:4554 , piece_id=46GFR7YBSFWWZ4FKKH7NRRIKWHPUKM2SV5CPVJQVYJV4YOOCYXEQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.490160132 (516h5m4.361243177s) [noise=true , address=145.79.227.10:4401 , piece_id=JMKNAW2HKBMMVNXR3R4R36OY4MKPKYLYJ2DD7ZMYH7VSQODIRHTA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.490724516 (516h5m4.361807561s) [noise=true , address=145.79.227.49:4339 , piece_id=D5V5NWUBUFL2CHUPRITXISTUABMN4IKWBPKXDQMZN2U3MFL3SGFQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.491214810 (516h5m4.362297855s) [noise=true , address=145.79.227.11:4731 , piece_id=7XK2P2JTDR7UTDQODNJ3JSC6Q66NPJAIUYLDXSM3M5HZOSYY4FGA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.491977733 (516h5m4.363060768s) [noise=true , address=145.79.227.9:4365 , piece_id=6X5CNCUDD3VZJWBEUP7IB76XWO6WXXTY6I4JE7WAEYSG6U7E2LKQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.492564941 (516h5m4.363647986s) [noise=true , address=45.91.116.7:4010 , piece_id=57WNHJAQWGNICJ5GU2I6MTVWOFH2C6A67LF3SWA47U56MBBWVBZA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.493203927 (516h5m4.364286972s) [noise=true , address=77.90.31.15:4700 , piece_id=D4J2A44AKEVYYAT3UT5Y57DOKCEE4ISTO27EDF4FDB2KRWSSA4WA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.494295506 (516h5m4.365378551s) [noise=true , address=145.79.227.43:4158 , piece_id=L2OG5DUNLCLYQOAPCURAEJ2N2HXHKXLTDYZSPZ2KYB5JOSSBUWAA, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:04.494913162 (516h5m4.365996207s) [noise=true , address=45.91.116.10:4081 , piece_id=ZB4XT6LXC7GQQOJQ5IPEMWAHE7LCJV3ISQEU455ZCEYANZUU3EXQ, seg_pos={0 0}, upload=452830]: Uploading piece... Dec 27 03:34:06.107398013 (516h5m5.978481048s) [upload=452830]: Next returned nil segment Dec 27 03:34:06.107411869 (516h5m5.978494904s) [upload=452830]: Waiting for error group managing segments... Dec 27 03:34:06.495613799 (516h5m6.366696834s) [noise=true , address=145.79.227.10:4401 , piece_id=JMKNAW2HKBMMVNXR3R4R36OY4MKPKYLYJ2DD7ZMYH7VSQODIRHTA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.496075158 (516h5m6.367158203s) [noise=true , address=145.79.227.9:4365 , piece_id=6X5CNCUDD3VZJWBEUP7IB76XWO6WXXTY6I4JE7WAEYSG6U7E2LKQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.501918784 (516h5m6.373001829s) [noise=true , address=145.79.227.45:4231 , piece_id=7AVKMKZAXMD7KNYJFV6BN6KT5NSWXSIWJYDLIY3AHMWAGVNMVQIQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.502537422 (516h5m6.373620468s) [noise=true , address=145.79.227.37:4566 , piece_id=BIIGCSDWIXW7M462LUQCZSOIWPC4DZDCDCLDWXJOPI445LZPD26A, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.510486179 (516h5m6.381569224s) [noise=true , address=145.79.227.52:4446 , piece_id=KKPOCFIESLU35ZBHJCY5O25JBWUV6IZAUYIMJ6QYHKANFRI7DITQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.518088785 (516h5m6.38917182s) [noise=true , address=145.79.227.11:4731 , piece_id=7XK2P2JTDR7UTDQODNJ3JSC6Q66NPJAIUYLDXSM3M5HZOSYY4FGA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.520758940 (516h5m6.391841985s) [noise=true , address=145.79.227.36:4519 , piece_id=WEAKNDM3ZCPZ6JOK6FTKKLV2GUSDGKPL24V5AY4XHXN6YXLOCB3A, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.530331888 (516h5m6.401414934s) [noise=true , address=145.79.227.49:4339 , piece_id=D5V5NWUBUFL2CHUPRITXISTUABMN4IKWBPKXDQMZN2U3MFL3SGFQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.679953787 (516h5m6.551036832s) [noise=true , address=77.90.31.13:4627 , piece_id=HWHEVCN7OU6N343CCUYQ463OVJX7VE2WY4JWGPST4IVSAD746SDA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.777916247 (516h5m6.648999282s) [noise=true , address=45.91.116.7:4010 , piece_id=57WNHJAQWGNICJ5GU2I6MTVWOFH2C6A67LF3SWA47U56MBBWVBZA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.797929275 (516h5m6.66901231s) [noise=true , address=77.90.31.11:4554 , piece_id=46GFR7YBSFWWZ4FKKH7NRRIKWHPUKM2SV5CPVJQVYJV4YOOCYXEQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:06.931457328 (516h5m6.802540373s) [noise=true , address=45.91.116.10:4081 , piece_id=ZB4XT6LXC7GQQOJQ5IPEMWAHE7LCJV3ISQEU455ZCEYANZUU3EXQ, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:07.024167666 (516h5m6.895250711s) [noise=true , address=45.91.116.12:4225 , piece_id=VPYHHXTTZOX5WFCY2YDC5PAF4A5HWKA7TJY2BCOMQ6MLR47LULBA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: <nil> Dec 27 03:34:07.024208094 (516h5m6.895291129s) [seg_pos={0 0}, upload=452830]: Segment reached optimal threshold of 13 pieces. Dec 27 03:34:07.024474891 (516h5m6.895557926s) [noise=true , address=77.90.31.15:4700 , piece_id=D4J2A44AKEVYYAT3UT5Y57DOKCEE4ISTO27EDF4FDB2KRWSSA4WA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:56044->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:56044->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56044->77.90.31.15:4700: use of closed network connection Dec 27 03:34:07.024491623 (516h5m6.895574658s) [noise=true , address=77.90.31.15:4700 , piece_id=D4J2A44AKEVYYAT3UT5Y57DOKCEE4ISTO27EDF4FDB2KRWSSA4WA, seg_pos={0 0}, upload=452830]: Long tail cancellation due to optimal threshold Dec 27 03:34:07.024565113 (516h5m6.895648158s) [noise=true , address=77.90.31.10:4520 , piece_id=43XIRJESKJMQFRWCZ5RWCE6K4RLJWOXSA7LSHSEZNSFQ7XUPSC4A, seg_pos={0 0}, upload=452830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 03:34:07.024573990 (516h5m6.895657025s) [noise=true , address=77.90.31.10:4520 , piece_id=43XIRJESKJMQFRWCZ5RWCE6K4RLJWOXSA7LSHSEZNSFQ7XUPSC4A, seg_pos={0 0}, upload=452830]: Long tail cancellation due to optimal threshold Dec 27 03:34:07.024755435 (516h5m6.89583848s) [noise=true , address=145.79.227.43:4158 , piece_id=L2OG5DUNLCLYQOAPCURAEJ2N2HXHKXLTDYZSPZ2KYB5JOSSBUWAA, seg_pos={0 0}, upload=452830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 03:34:07.024768240 (516h5m6.895851285s) [noise=true , address=145.79.227.43:4158 , piece_id=L2OG5DUNLCLYQOAPCURAEJ2N2HXHKXLTDYZSPZ2KYB5JOSSBUWAA, seg_pos={0 0}, upload=452830]: Long tail cancellation due to optimal threshold Dec 27 03:34:07.024813075 (516h5m6.89589612s) [seg_pos={0 0}, upload=452830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:34:07.024909539 (516h5m6.895992574s) [upload=452830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:34:07.024914609 (516h5m6.895997644s) [upload=452830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:34:07.316939289 (516h5m7.188022324s) [upload=452830]: Done uploading segments... Dec 27 03:34:07.316964897 (516h5m7.188047942s) [upload=452830]: Upload finished. err: <nil> Duration: 30.00271918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XXX] duration: 30.004295259s trace: https://jaeger.dev.storj.io/trace/d03f09a85c484c FAIL (2025-12-27 03:34:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: Y7ZYXYMS6ERKJVXBOWZO5WBCJIDZRNQ4CUTI7AVWPRZXIUBTSOKA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: CQPVMDHJOJX46OKQIIFXJJCCIM4H26GW6VXXP3GEWQVH4EULIQFA) context deadline exceeded ======LOGS====== Dec 27 03:33:34.003469872 (516h4m33.874552907s) [upload=452822]: Starting upload Dec 27 03:33:34.003536879 (516h4m33.874619914s) [upload=452822]: Done starting upload Dec 27 03:33:34.003620067 (516h4m33.874703112s) [upload=452822]: Uploading segments... Dec 27 03:33:34.003760314 (516h4m33.874843359s) [upload=452822]: Got next segment. Inline: false Dec 27 03:33:34.003796414 (516h4m33.874879459s) [upload=452822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:33:34.003801474 (516h4m33.874884509s) [upload=452822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:33:34.253891837 (516h4m34.124974873s) [seg_pos={0 0}, upload=452822]: Begin upload segment... Dec 27 03:33:34.253944327 (516h4m34.125027362s) [seg_pos={0 0}, upload=452822]: Done begin upload segment. Dec 27 03:33:34.253972090 (516h4m34.125055135s) [noise=true , address=145.79.227.51:4409 , piece_id=AEMQSGI5MBNS5DO34OPX5P3LC3GI2ACEQ5WQABXSHIIW7MEDB7OQ, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.253986267 (516h4m34.125069312s) [noise=true , address=145.79.227.42:4122 , piece_id=DDL27FW5PU6GTKBPCCUEFGH5IHDQVQ3MZI2YFWNDC2SI3B37RTTA, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.254401988 (516h4m34.125485043s) [noise=true , address=145.79.227.24:4582 , piece_id=ZOKJNMBP7QTCXHSMH4766LECWX6XXYRNNLUO6GDHYAVKIMVJIYAQ, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.254861363 (516h4m34.125944408s) [noise=true , address=145.79.227.48:4303 , piece_id=RLLHUWXB2TWRUUFNCUGHW6MNII262PLS5RY5D4EJNENM4M3IPVPQ, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.255245876 (516h4m34.126328911s) [noise=true , address=145.79.227.17:4971 , piece_id=QK64RT6JLSEH7QOLXXXCLXTBFT5G4O56YRKWDD5NB2OBVHJNQKKA, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.256252772 (516h4m34.127335817s) [noise=true , address=77.90.31.21:4880 , piece_id=KWSG2OC4NQZDQVK3ZZKVWWYWM5GVYZA73HLH3TMUB7PS2GVOETHA, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.256895116 (516h4m34.127978161s) [noise=true , address=145.79.227.4:4185 , piece_id=PIU5FZWA4UUEJVZXT4QI7PKRLIA55UGYCJI5XG4MWM7CYPZX6S2A, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.258364884 (516h4m34.12944793s) [noise=true , address=45.91.116.12:4225 , piece_id=Y7ZYXYMS6ERKJVXBOWZO5WBCJIDZRNQ4CUTI7AVWPRZXIUBTSOKA, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.263096715 (516h4m34.13417975s) [noise=true , address=45.91.116.13:4154 , piece_id=NKILIVCZRPL2LSDGA6XRH3V54F47GFFQNUYG7XFAKIA6MNHAWZTA, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.265834699 (516h4m34.136917744s) [noise=true , address=45.91.116.11:4119 , piece_id=K6ZIJ2ATKNLNPUGSSZBK7OLHJ2ST4WCYOVUTPUQNL7SEVBJFHMBQ, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.268536443 (516h4m34.139619488s) [noise=true , address=45.91.116.14:4191 , piece_id=CQPVMDHJOJX46OKQIIFXJJCCIM4H26GW6VXXP3GEWQVH4EULIQFA, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.270199610 (516h4m34.141282645s) [noise=true , address=145.79.227.6:4259 , piece_id=P2CU4FDGMBSKZZELXH6IJ5IXPK6N67RS6JFG4ZL4Z5OM7XBADJGA, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.271932109 (516h4m34.143015204s) [noise=true , address=145.79.227.15:4874 , piece_id=K7FUMF2MDK7D7JYPXJNKM5KHTV7NG3HJSFIEJ7C7J3JQG2Q7Q6BA, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.273447434 (516h4m34.144530469s) [noise=true , address=145.79.227.46:4601 , piece_id=XXKWT73BCUEEMOLHS5XKS35MG3TJ5H2O67OJWAQC5R3KJ476KWUQ, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.274137919 (516h4m34.145220954s) [noise=true , address=145.79.227.36:4518 , piece_id=AB2G5QMIESWXFQ7CIV7VIIB32OL3ADC3V2XUEYRJL6PTHMLUAZAQ, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:34.276042947 (516h4m34.147125982s) [noise=true , address=145.79.227.34:4907 , piece_id=UWV5L5IR3RRT6H7D24IUFUAKARLMGLXC4HQH7DBOOVOTZT4AY3CQ, seg_pos={0 0}, upload=452822]: Uploading piece... Dec 27 03:33:35.964788960 (516h4m35.835871995s) [upload=452822]: Next returned nil segment Dec 27 03:33:35.964830700 (516h4m35.835913735s) [upload=452822]: Waiting for error group managing segments... Dec 27 03:33:36.248192666 (516h4m36.119275701s) [noise=true , address=145.79.227.24:4582 , piece_id=ZOKJNMBP7QTCXHSMH4766LECWX6XXYRNNLUO6GDHYAVKIMVJIYAQ, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.257960576 (516h4m36.129043621s) [noise=true , address=145.79.227.51:4409 , piece_id=AEMQSGI5MBNS5DO34OPX5P3LC3GI2ACEQ5WQABXSHIIW7MEDB7OQ, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.260141949 (516h4m36.131224984s) [noise=true , address=145.79.227.4:4185 , piece_id=PIU5FZWA4UUEJVZXT4QI7PKRLIA55UGYCJI5XG4MWM7CYPZX6S2A, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.260719188 (516h4m36.131802224s) [noise=true , address=145.79.227.48:4303 , piece_id=RLLHUWXB2TWRUUFNCUGHW6MNII262PLS5RY5D4EJNENM4M3IPVPQ, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.266044389 (516h4m36.137127424s) [noise=true , address=145.79.227.42:4122 , piece_id=DDL27FW5PU6GTKBPCCUEFGH5IHDQVQ3MZI2YFWNDC2SI3B37RTTA, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.268577613 (516h4m36.139660648s) [noise=true , address=145.79.227.17:4971 , piece_id=QK64RT6JLSEH7QOLXXXCLXTBFT5G4O56YRKWDD5NB2OBVHJNQKKA, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.282496607 (516h4m36.153579652s) [noise=true , address=145.79.227.6:4259 , piece_id=P2CU4FDGMBSKZZELXH6IJ5IXPK6N67RS6JFG4ZL4Z5OM7XBADJGA, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.288664441 (516h4m36.159747476s) [noise=true , address=145.79.227.15:4874 , piece_id=K7FUMF2MDK7D7JYPXJNKM5KHTV7NG3HJSFIEJ7C7J3JQG2Q7Q6BA, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.295999188 (516h4m36.167082263s) [noise=true , address=145.79.227.46:4601 , piece_id=XXKWT73BCUEEMOLHS5XKS35MG3TJ5H2O67OJWAQC5R3KJ476KWUQ, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.303419367 (516h4m36.174502412s) [noise=true , address=145.79.227.36:4518 , piece_id=AB2G5QMIESWXFQ7CIV7VIIB32OL3ADC3V2XUEYRJL6PTHMLUAZAQ, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.506047931 (516h4m36.377130976s) [noise=true , address=45.91.116.12:4225 , piece_id=Y7ZYXYMS6ERKJVXBOWZO5WBCJIDZRNQ4CUTI7AVWPRZXIUBTSOKA, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:36.572477305 (516h4m36.44356034s) [noise=true , address=77.90.31.21:4880 , piece_id=KWSG2OC4NQZDQVK3ZZKVWWYWM5GVYZA73HLH3TMUB7PS2GVOETHA, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:37.177435132 (516h4m37.048518177s) [noise=true , address=45.91.116.14:4191 , piece_id=CQPVMDHJOJX46OKQIIFXJJCCIM4H26GW6VXXP3GEWQVH4EULIQFA, seg_pos={0 0}, upload=452822]: Done uploading piece. err: <nil> Dec 27 03:33:37.177459557 (516h4m37.048542602s) [seg_pos={0 0}, upload=452822]: Segment reached optimal threshold of 13 pieces. Dec 27 03:33:37.177711468 (516h4m37.048794523s) [noise=true , address=145.79.227.34:4907 , piece_id=UWV5L5IR3RRT6H7D24IUFUAKARLMGLXC4HQH7DBOOVOTZT4AY3CQ, seg_pos={0 0}, upload=452822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 03:33:37.177726256 (516h4m37.048809301s) [noise=true , address=145.79.227.34:4907 , piece_id=UWV5L5IR3RRT6H7D24IUFUAKARLMGLXC4HQH7DBOOVOTZT4AY3CQ, seg_pos={0 0}, upload=452822]: Long tail cancellation due to optimal threshold Dec 27 03:33:37.177813932 (516h4m37.048896977s) [noise=true , address=45.91.116.13:4154 , piece_id=NKILIVCZRPL2LSDGA6XRH3V54F47GFFQNUYG7XFAKIA6MNHAWZTA, seg_pos={0 0}, upload=452822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 03:33:37.177830183 (516h4m37.048913219s) [noise=true , address=45.91.116.13:4154 , piece_id=NKILIVCZRPL2LSDGA6XRH3V54F47GFFQNUYG7XFAKIA6MNHAWZTA, seg_pos={0 0}, upload=452822]: Long tail cancellation due to optimal threshold Dec 27 03:33:37.177954100 (516h4m37.049037145s) [noise=true , address=45.91.116.11:4119 , piece_id=K6ZIJ2ATKNLNPUGSSZBK7OLHJ2ST4WCYOVUTPUQNL7SEVBJFHMBQ, seg_pos={0 0}, upload=452822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 03:33:37.177962466 (516h4m37.049045511s) [noise=true , address=45.91.116.11:4119 , piece_id=K6ZIJ2ATKNLNPUGSSZBK7OLHJ2ST4WCYOVUTPUQNL7SEVBJFHMBQ, seg_pos={0 0}, upload=452822]: Long tail cancellation due to optimal threshold Dec 27 03:33:37.177982142 (516h4m37.049065188s) [seg_pos={0 0}, upload=452822]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:33:37.178144322 (516h4m37.049227367s) [upload=452822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:33:37.178151195 (516h4m37.04923424s) [upload=452822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:33:37.479333133 (516h4m37.350416198s) [upload=452822]: Done uploading segments... Dec 27 03:33:37.479353581 (516h4m37.350436616s) [upload=452822]: Upload finished. err: <nil> Duration: 30.004318463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.XX] duration: 30.002499942s trace: https://jaeger.dev.storj.io/trace/2204d896f9def361 FAIL (2025-12-27 03:32:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 03:32:20.919947064 (516h3m20.791030109s) [upload=452807]: Starting upload Dec 27 03:32:20.920038930 (516h3m20.791121965s) [upload=452807]: Done starting upload Dec 27 03:32:20.920136095 (516h3m20.79121914s) [upload=452807]: Uploading segments... Dec 27 03:32:20.920296289 (516h3m20.791379324s) [upload=452807]: Got next segment. Inline: false Dec 27 03:32:20.920339953 (516h3m20.791422998s) [upload=452807]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:32:20.920345884 (516h3m20.791428929s) [upload=452807]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:32:21.168574594 (516h3m21.039657639s) [seg_pos={0 0}, upload=452807]: Begin upload segment... Dec 27 03:32:21.168638635 (516h3m21.03972167s) [seg_pos={0 0}, upload=452807]: Done begin upload segment. Dec 27 03:32:21.168666248 (516h3m21.039749293s) [noise=true , address=45.91.116.13:4155 , piece_id=TLGWTFUUJ3GPZ3NDYCXPDUVAYRI35CDPY4YPZLPTSSOO6OCKQLIQ, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.168796746 (516h3m21.039879781s) [noise=true , address=145.79.227.39:4014 , piece_id=2XYPFF7STNFHXJCXIJPTNY7IIVUSVSCOXQ6XQ64HAT2QPFFJITJA, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.169338087 (516h3m21.040421132s) [noise=true , address=77.90.31.10:4519 , piece_id=MX44QX33G3ZUMEQPZKVG3A7ZNXYGPLSXIW5OOX4RM76MMRKMXFDQ, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.169896110 (516h3m21.040979155s) [noise=true , address=77.90.31.12:4589 , piece_id=H7V56INNDACVF66D7CT46PLO4L32Z6BO5NCDH6FUQF2H4SVMTILQ, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.170404337 (516h3m21.041487372s) [noise=true , address=145.79.227.37:4566 , piece_id=MGM3XHJUST7KTEJGNJYPZ6TBJ2TA66TNNEUCJ5JAC3P4RYRRMUIA, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.171023216 (516h3m21.042106261s) [noise=true , address=145.79.227.15:4873 , piece_id=MB4JGBKLMGXVFJKEECPZTIJLMVWIPMIQDC37UDBFPSUDWDRSL2WQ, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.171556291 (516h3m21.042639336s) [noise=true , address=145.79.227.16:4911 , piece_id=CIBQB3FV4XXUHQOOQVXZIXP3QOAZWEVMMO7JG3VVG2XHXFUPJ74A, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.173093738 (516h3m21.044176783s) [noise=true , address=145.79.227.3:4151 , piece_id=DHL7FHDXAUPCUAG36WMEEUMDXXL6TSHOXYEYCG37UZTCJPZ3C4KQ, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.173336000 (516h3m21.044419045s) [noise=true , address=77.90.31.11:4554 , piece_id=3G7R4545YKMTUCE2KEO6SWPTRKSFQGYIXHC5R5KDIYQIMSL7L6EA, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.175067948 (516h3m21.046150993s) [noise=true , address=77.90.31.16:4736 , piece_id=3YQ2NUCVPPSETJCXWUJSTORPBI2ATFQYNDVJSUAYSMDABYUU5LUQ, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.175175532 (516h3m21.046258578s) [noise=true , address=45.91.116.6:4976 , piece_id=EHLWFXPLOWSWBFBOLEHVZVCJQQWQUV2IABN4OVNBSCGJLNOWID4A, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.177147748 (516h3m21.048230793s) [noise=true , address=45.91.116.9:4263 , piece_id=4NFNDIZOZB6LQ53XMJGQPPV4F55XCSKXZAG4BEXOIUYQRO4Z7GPQ, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.177209636 (516h3m21.048292671s) [noise=true , address=45.91.116.8:4048 , piece_id=4HWDJ4ISVDA7RWBQNW42KGD3BZMCTCIC4XPMUXOSLO6LDEYY5Q6Q, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.178928168 (516h3m21.050011203s) [noise=true , address=145.79.227.11:4731 , piece_id=PLQ2L43I25ET52W7TY4MYY7BSGN2YML6LXHXT5D2DMUGFF3WIABA, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.179350232 (516h3m21.050433277s) [noise=true , address=45.91.116.7:4009 , piece_id=UI3PZ35SSWLAJZWK42LXHIH6LUXWXZL5P2Q64ZGB7U4AVDMRIU4Q, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:21.183592009 (516h3m21.054675054s) [noise=true , address=145.79.227.44:4194 , piece_id=C7BRSW2EXBKPUMETQ5J7BJHPTO5KATVQKXBVPDZLYBYHFPZQMAIA, seg_pos={0 0}, upload=452807]: Uploading piece... Dec 27 03:32:22.802574465 (516h3m22.6736575s) [upload=452807]: Next returned nil segment Dec 27 03:32:22.802595175 (516h3m22.67367822s) [upload=452807]: Waiting for error group managing segments... Dec 27 03:32:23.192583558 (516h3m23.063666593s) [noise=true , address=145.79.227.11:4731 , piece_id=PLQ2L43I25ET52W7TY4MYY7BSGN2YML6LXHXT5D2DMUGFF3WIABA, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.199287885 (516h3m23.07037093s) [noise=true , address=145.79.227.3:4151 , piece_id=DHL7FHDXAUPCUAG36WMEEUMDXXL6TSHOXYEYCG37UZTCJPZ3C4KQ, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.202987890 (516h3m23.074070935s) [noise=true , address=145.79.227.15:4873 , piece_id=MB4JGBKLMGXVFJKEECPZTIJLMVWIPMIQDC37UDBFPSUDWDRSL2WQ, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.203140411 (516h3m23.074223456s) [noise=true , address=145.79.227.39:4014 , piece_id=2XYPFF7STNFHXJCXIJPTNY7IIVUSVSCOXQ6XQ64HAT2QPFFJITJA, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.204923406 (516h3m23.076006451s) [noise=true , address=145.79.227.37:4566 , piece_id=MGM3XHJUST7KTEJGNJYPZ6TBJ2TA66TNNEUCJ5JAC3P4RYRRMUIA, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.221077165 (516h3m23.09216021s) [noise=true , address=145.79.227.16:4911 , piece_id=CIBQB3FV4XXUHQOOQVXZIXP3QOAZWEVMMO7JG3VVG2XHXFUPJ74A, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.435132963 (516h3m23.306215998s) [noise=true , address=45.91.116.8:4048 , piece_id=4HWDJ4ISVDA7RWBQNW42KGD3BZMCTCIC4XPMUXOSLO6LDEYY5Q6Q, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.450703272 (516h3m23.321786317s) [noise=true , address=77.90.31.11:4554 , piece_id=3G7R4545YKMTUCE2KEO6SWPTRKSFQGYIXHC5R5KDIYQIMSL7L6EA, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.485511102 (516h3m23.356594157s) [noise=true , address=77.90.31.16:4736 , piece_id=3YQ2NUCVPPSETJCXWUJSTORPBI2ATFQYNDVJSUAYSMDABYUU5LUQ, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.518550225 (516h3m23.38963326s) [noise=true , address=45.91.116.7:4009 , piece_id=UI3PZ35SSWLAJZWK42LXHIH6LUXWXZL5P2Q64ZGB7U4AVDMRIU4Q, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.557396354 (516h3m23.428479389s) [noise=true , address=77.90.31.10:4519 , piece_id=MX44QX33G3ZUMEQPZKVG3A7ZNXYGPLSXIW5OOX4RM76MMRKMXFDQ, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.598998251 (516h3m23.470081296s) [noise=true , address=45.91.116.6:4976 , piece_id=EHLWFXPLOWSWBFBOLEHVZVCJQQWQUV2IABN4OVNBSCGJLNOWID4A, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.930805208 (516h3m23.801888243s) [noise=true , address=45.91.116.9:4263 , piece_id=4NFNDIZOZB6LQ53XMJGQPPV4F55XCSKXZAG4BEXOIUYQRO4Z7GPQ, seg_pos={0 0}, upload=452807]: Done uploading piece. err: <nil> Dec 27 03:32:23.930830846 (516h3m23.801913891s) [seg_pos={0 0}, upload=452807]: Segment reached optimal threshold of 13 pieces. Dec 27 03:32:23.931063269 (516h3m23.802146304s) [noise=true , address=77.90.31.12:4589 , piece_id=H7V56INNDACVF66D7CT46PLO4L32Z6BO5NCDH6FUQF2H4SVMTILQ, seg_pos={0 0}, upload=452807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:38694->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:38694->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38694->77.90.31.12:4589: use of closed network connection Dec 27 03:32:23.931071174 (516h3m23.802154209s) [noise=true , address=77.90.31.12:4589 , piece_id=H7V56INNDACVF66D7CT46PLO4L32Z6BO5NCDH6FUQF2H4SVMTILQ, seg_pos={0 0}, upload=452807]: Long tail cancellation due to optimal threshold Dec 27 03:32:23.931145847 (516h3m23.802228882s) [noise=true , address=45.91.116.13:4155 , piece_id=TLGWTFUUJ3GPZ3NDYCXPDUVAYRI35CDPY4YPZLPTSSOO6OCKQLIQ, seg_pos={0 0}, upload=452807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 03:32:23.931171134 (516h3m23.802254169s) [noise=true , address=45.91.116.13:4155 , piece_id=TLGWTFUUJ3GPZ3NDYCXPDUVAYRI35CDPY4YPZLPTSSOO6OCKQLIQ, seg_pos={0 0}, upload=452807]: Long tail cancellation due to optimal threshold Dec 27 03:32:23.931339415 (516h3m23.80242245s) [noise=true , address=145.79.227.44:4194 , piece_id=C7BRSW2EXBKPUMETQ5J7BJHPTO5KATVQKXBVPDZLYBYHFPZQMAIA, seg_pos={0 0}, upload=452807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 03:32:23.931347390 (516h3m23.802430425s) [noise=true , address=145.79.227.44:4194 , piece_id=C7BRSW2EXBKPUMETQ5J7BJHPTO5KATVQKXBVPDZLYBYHFPZQMAIA, seg_pos={0 0}, upload=452807]: Long tail cancellation due to optimal threshold Dec 27 03:32:23.931396143 (516h3m23.802479188s) [seg_pos={0 0}, upload=452807]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:32:23.931484841 (516h3m23.802567886s) [upload=452807]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:32:23.931491474 (516h3m23.802574519s) [upload=452807]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:32:24.222913431 (516h3m24.093996466s) [upload=452807]: Done uploading segments... Dec 27 03:32:24.222933830 (516h3m24.094016865s) [upload=452807]: Upload finished. err: <nil> Duration: 30.002519929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.X] duration: 30.002967674s trace: https://jaeger.dev.storj.io/trace/5e058138840a0759 FAIL (2025-12-27 03:31:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6h2m28.068148803s) [upload=452792]: Uploading segments... Dec 27 03:31:28.197175747 (516h2m28.068258782s) [upload=452792]: Got next segment. Inline: false Dec 27 03:31:28.197203611 (516h2m28.068286646s) [upload=452792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:31:28.197207949 (516h2m28.068290984s) [upload=452792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:31:28.438992087 (516h2m28.310075122s) [seg_pos={0 0}, upload=452792]: Begin upload segment... Dec 27 03:31:28.439039357 (516h2m28.310122393s) [seg_pos={0 0}, upload=452792]: Done begin upload segment. Dec 27 03:31:28.439066369 (516h2m28.310149414s) [noise=true , address=145.79.227.29:4943 , piece_id=TALIVEAINTDREAXPNCWMSVAHTD3BHES754Y7BQUHQHZXBISVP7MQ, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.439786229 (516h2m28.310869274s) [noise=true , address=145.79.227.6:4259 , piece_id=D3M5VRVGGNROXZHJZQYLDFO472YBA54MC5WCVFTCXXOTVRFK3JSA, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.440116058 (516h2m28.311199093s) [noise=true , address=145.79.227.34:4905 , piece_id=7LXYGSFE7RSBDXFSSISC2IC4KWD3AXBSBUB7RYBIOSHPLOCWJZSA, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.440421388 (516h2m28.311504433s) [noise=true , address=45.91.116.5:4297 , piece_id=YYXDIFDYCVSUO7QPKZ5FPISB3MNSFIMRPWWTXFBPR4A2OTDZ3FKQ, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.440698366 (516h2m28.311781401s) [noise=true , address=145.79.227.16:4911 , piece_id=VT2XW2AST5IFLFOZMCIU23NHX5YAVQFE6UHKYAN3HRO4F3YY3ZYA, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.440968811 (516h2m28.312051856s) [noise=true , address=145.79.227.52:4445 , piece_id=CLO5IQ4DMRYNAWWXBWMKKOILTJJLPSP25JVUCINO2UIKAKT5XYFQ, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.441294582 (516h2m28.312377617s) [noise=true , address=145.79.227.9:4366 , piece_id=KBQQCICCVTP6RKOTXTNYZXCB4XU5PXVEJUICPJMBJLNY5XT7JTDQ, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.441672040 (516h2m28.312755095s) [noise=true , address=45.91.116.10:4084 , piece_id=4Z4OF2KYM6UYWC5IR7JCYF5R3OLA7XJMWFMAYUDAZ6AUFL4OQYKA, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.443194529 (516h2m28.314277574s) [noise=true , address=77.90.31.11:4556 , piece_id=WAGY3JCUREKNC57JIEPR3JVDJO2J2VTAJIDDA43E3XSIY37LHC4A, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.444447385 (516h2m28.31553043s) [noise=true , address=45.91.116.11:4119 , piece_id=ROWIHGY5XTX2PHXHXWTWVG4DM7DJ3ZHHNIFP2NHMXC6S35H4FXEQ, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.445692255 (516h2m28.31677529s) [noise=true , address=77.90.31.21:4880 , piece_id=CK5I2B73FUFMWQQDEDWBC3632WIXTPRZBNJM5OLVSTIQYGQLI7TQ, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.446929291 (516h2m28.318012326s) [noise=true , address=145.79.227.15:4875 , piece_id=5TMZMW6C2ENLXRZTD356NXOALVWPJVZXCMU7AIQWQEJVKWVM2DAA, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.447488355 (516h2m28.31857139s) [noise=true , address=145.79.227.24:4582 , piece_id=HNHYHVXR2OYXJPCK3LRYLOGLKT3SPJDNTPKBK7DHICFY54G34BCA, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.448630640 (516h2m28.319713685s) [noise=true , address=145.79.227.11:4730 , piece_id=VIM6W536XO724NIIM2SXOW3L4FLG2K5RGFRJMNLOQITUAXQYG5OQ, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.449729993 (516h2m28.320813028s) [noise=true , address=145.79.227.17:4970 , piece_id=PXSRPTPF4MTFZXGCOXBIMPDIIDDWBUFF3EILZ5UXOFBU4CG3KQQA, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:28.450235897 (516h2m28.321318932s) [noise=true , address=45.91.116.6:4974 , piece_id=YPRK5AHILZIH45LN53JPHDKZTXEHGCUL7S5QNIDWATE5TX37UKYQ, seg_pos={0 0}, upload=452792]: Uploading piece... Dec 27 03:31:30.049745944 (516h2m29.92082898s) [upload=452792]: Next returned nil segment Dec 27 03:31:30.049757216 (516h2m29.920840261s) [upload=452792]: Waiting for error group managing segments... Dec 27 03:31:30.438846427 (516h2m30.309929462s) [noise=true , address=145.79.227.52:4445 , piece_id=CLO5IQ4DMRYNAWWXBWMKKOILTJJLPSP25JVUCINO2UIKAKT5XYFQ, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:30.454413449 (516h2m30.325496485s) [noise=true , address=145.79.227.11:4730 , piece_id=VIM6W536XO724NIIM2SXOW3L4FLG2K5RGFRJMNLOQITUAXQYG5OQ, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:30.455782327 (516h2m30.326865372s) [noise=true , address=145.79.227.15:4875 , piece_id=5TMZMW6C2ENLXRZTD356NXOALVWPJVZXCMU7AIQWQEJVKWVM2DAA, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:30.456546270 (516h2m30.327629316s) [noise=true , address=145.79.227.17:4970 , piece_id=PXSRPTPF4MTFZXGCOXBIMPDIIDDWBUFF3EILZ5UXOFBU4CG3KQQA, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:30.462212210 (516h2m30.333295255s) [noise=true , address=145.79.227.24:4582 , piece_id=HNHYHVXR2OYXJPCK3LRYLOGLKT3SPJDNTPKBK7DHICFY54G34BCA, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:30.465899120 (516h2m30.336982155s) [noise=true , address=145.79.227.34:4905 , piece_id=7LXYGSFE7RSBDXFSSISC2IC4KWD3AXBSBUB7RYBIOSHPLOCWJZSA, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:30.479047227 (516h2m30.350130272s) [noise=true , address=145.79.227.9:4366 , piece_id=KBQQCICCVTP6RKOTXTNYZXCB4XU5PXVEJUICPJMBJLNY5XT7JTDQ, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:30.497045589 (516h2m30.368128634s) [noise=true , address=145.79.227.16:4911 , piece_id=VT2XW2AST5IFLFOZMCIU23NHX5YAVQFE6UHKYAN3HRO4F3YY3ZYA, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:30.543805858 (516h2m30.414888893s) [noise=true , address=145.79.227.29:4943 , piece_id=TALIVEAINTDREAXPNCWMSVAHTD3BHES754Y7BQUHQHZXBISVP7MQ, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:30.663193502 (516h2m30.534276547s) [noise=true , address=77.90.31.21:4880 , piece_id=CK5I2B73FUFMWQQDEDWBC3632WIXTPRZBNJM5OLVSTIQYGQLI7TQ, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:30.803020206 (516h2m30.674103251s) [noise=true , address=45.91.116.11:4119 , piece_id=ROWIHGY5XTX2PHXHXWTWVG4DM7DJ3ZHHNIFP2NHMXC6S35H4FXEQ, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:31.426109783 (516h2m31.297192828s) [noise=true , address=45.91.116.10:4084 , piece_id=4Z4OF2KYM6UYWC5IR7JCYF5R3OLA7XJMWFMAYUDAZ6AUFL4OQYKA, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:31.547108776 (516h2m31.418191811s) [noise=true , address=77.90.31.11:4556 , piece_id=WAGY3JCUREKNC57JIEPR3JVDJO2J2VTAJIDDA43E3XSIY37LHC4A, seg_pos={0 0}, upload=452792]: Done uploading piece. err: <nil> Dec 27 03:31:31.547128024 (516h2m31.418211059s) [seg_pos={0 0}, upload=452792]: Segment reached optimal threshold of 13 pieces. Dec 27 03:31:31.547307305 (516h2m31.41839034s) [noise=true , address=145.79.227.6:4259 , piece_id=D3M5VRVGGNROXZHJZQYLDFO472YBA54MC5WCVFTCXXOTVRFK3JSA, seg_pos={0 0}, upload=452792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:60804->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:60804->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60804->145.79.227.6:4259: use of closed network connection Dec 27 03:31:31.547316131 (516h2m31.418399176s) [noise=true , address=145.79.227.6:4259 , piece_id=D3M5VRVGGNROXZHJZQYLDFO472YBA54MC5WCVFTCXXOTVRFK3JSA, seg_pos={0 0}, upload=452792]: Long tail cancellation due to optimal threshold Dec 27 03:31:31.547424247 (516h2m31.418507302s) [noise=true , address=45.91.116.5:4297 , piece_id=YYXDIFDYCVSUO7QPKZ5FPISB3MNSFIMRPWWTXFBPR4A2OTDZ3FKQ, seg_pos={0 0}, upload=452792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 27 03:31:31.547469394 (516h2m31.418552439s) [noise=true , address=45.91.116.6:4974 , piece_id=YPRK5AHILZIH45LN53JPHDKZTXEHGCUL7S5QNIDWATE5TX37UKYQ, seg_pos={0 0}, upload=452792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:54266->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:54266->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54266->45.91.116.6:4974: use of closed network connection Dec 27 03:31:31.547479753 (516h2m31.418562788s) [noise=true , address=45.91.116.6:4974 , piece_id=YPRK5AHILZIH45LN53JPHDKZTXEHGCUL7S5QNIDWATE5TX37UKYQ, seg_pos={0 0}, upload=452792]: Long tail cancellation due to optimal threshold Dec 27 03:31:31.547516573 (516h2m31.418599618s) [noise=true , address=45.91.116.5:4297 , piece_id=YYXDIFDYCVSUO7QPKZ5FPISB3MNSFIMRPWWTXFBPR4A2OTDZ3FKQ, seg_pos={0 0}, upload=452792]: Long tail cancellation due to optimal threshold Dec 27 03:31:31.547642472 (516h2m31.418725517s) [seg_pos={0 0}, upload=452792]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:31:31.547817647 (516h2m31.418900692s) [upload=452792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:31:31.547857352 (516h2m31.418940397s) [upload=452792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:31:31.829937761 (516h2m31.701020836s) [upload=452792]: Done uploading segments... Dec 27 03:31:31.829963129 (516h2m31.701046185s) [upload=452792]: Upload finished. err: <nil> Duration: 30.002994073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX.] duration: 6.093880549s trace: https://jaeger.dev.storj.io/trace/68cfb97b8a21353f FAIL (2025-12-27 03:30:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: 7AIN6YI6RJZK6DLYVOZT5HRVAZDUEIWQJEYY6N5GJ7QS2YMXLBCA) piecestore close: write tcp 188.166.225.170:56336->145.79.227.2:4113: use of closed network connection; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZJNU5RHMFKZ2QYEHCZQQ56IVIMV5TEDCXEHNFBLJIFK5T4KGSKZQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: KXLM7LNJDFHJPJJPRAS6MJPUX2G52WBCLKM2GEOSIYXGAJAQNYZA) context deadline exceeded ======LOGS====== Dec 27 03:29:20.775216031 (516h0m20.646299066s) [upload=452760]: Starting upload Dec 27 03:29:20.775289100 (516h0m20.646372135s) [upload=452760]: Done starting upload Dec 27 03:29:20.775418817 (516h0m20.646501852s) [upload=452760]: Uploading segments... Dec 27 03:29:20.775540017 (516h0m20.646623052s) [upload=452760]: Got next segment. Inline: false Dec 27 03:29:20.775589933 (516h0m20.646672968s) [upload=452760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:29:20.775595163 (516h0m20.646678198s) [upload=452760]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:29:21.020329299 (516h0m20.891412344s) [seg_pos={0 0}, upload=452760]: Begin upload segment... Dec 27 03:29:21.020375938 (516h0m20.891458983s) [seg_pos={0 0}, upload=452760]: Done begin upload segment. Dec 27 03:29:21.020403511 (516h0m20.891486556s) [noise=true , address=145.79.227.51:4410 , piece_id=5QJ2HOEEL2HXXQ6TBW43UIPZX3SSJDUCUE2JPE6CTQF7AVYIJYBQ, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.020465458 (516h0m20.891548503s) [noise=true , address=145.79.227.5:4221 , piece_id=B2QTSVBQDLEIEYT7XKHJAFGETP34SZDDYFHHWXGVSD7OU4ZDONSA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.021059820 (516h0m20.892142865s) [noise=true , address=145.79.227.38:4977 , piece_id=QBCPI2XH5Q7JFME77RTUXARSE3T2CF2R4GERNTLW2Y5DGYD275GA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.021462236 (516h0m20.892545281s) [noise=true , address=145.79.227.2:4113 , piece_id=7AIN6YI6RJZK6DLYVOZT5HRVAZDUEIWQJEYY6N5GJ7QS2YMXLBCA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.021916080 (516h0m20.892999115s) [noise=true , address=77.90.31.10:4519 , piece_id=ZJNU5RHMFKZ2QYEHCZQQ56IVIMV5TEDCXEHNFBLJIFK5T4KGSKZQ, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.022296044 (516h0m20.893379089s) [noise=true , address=45.91.116.4:4939 , piece_id=FYX6EVAOIELVDH3WVROJQLJF7QH3OKVNC2K55QAQCOSBLUPVQRDA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.022777030 (516h0m20.893860065s) [noise=true , address=145.79.227.52:4447 , piece_id=4L3EUSG5TRSYNXQMHIC47O5P6CRGSZQ4NFE7NR2V6UADGF7PC7MQ, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.023593003 (516h0m20.894676038s) [noise=true , address=45.91.116.11:4120 , piece_id=KXLM7LNJDFHJPJJPRAS6MJPUX2G52WBCLKM2GEOSIYXGAJAQNYZA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.023657326 (516h0m20.894740371s) [noise=true , address=77.90.31.14:4661 , piece_id=CIFUDAKEQ5WZ54LXT4J2QOSS7PD4RVWB6B5LREDWLEYYC4NYIGZA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.024223995 (516h0m20.89530704s) [noise=true , address=145.79.227.46:4603 , piece_id=EP7YIZBFN2VYWTDPKTDI2PCIWLJLDALQAHIZWCSRGR546ARYDCYA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.024566888 (516h0m20.895649933s) [noise=true , address=45.91.116.9:4262 , piece_id=7263XABPRAG4L7VYNRXMUP4ZW2CPWNFFFPPGRAXGU6C3ZULRRS3A, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.025615274 (516h0m20.896698309s) [noise=true , address=45.91.116.14:4191 , piece_id=YRFMS3A2X4OQQA3DQQQFCVFVZI6HLBSKHUQ4U7F5XXSN27M7O42Q, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.026613816 (516h0m20.897696861s) [noise=true , address=45.91.116.10:4084 , piece_id=4LPIECHH5RFVH74J5BGN3THENYSGPHLAPNL3Q5WFFRMY3Q7UCXKA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.027385194 (516h0m20.898468229s) [noise=true , address=77.90.31.13:4627 , piece_id=CS3FQ44IQAPZFINZ25A3NGFUGHUFNNSH3ZBS4JFUHONWX2NLXK5Q, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.027723087 (516h0m20.898806122s) [noise=true , address=145.79.227.13:4804 , piece_id=GF22Z64EUQPRBMXVLFDE5HXCNZQCCQZYN2EFHOE3GWEV6HOUUMUA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.028729223 (516h0m20.899812268s) [noise=true , address=145.79.227.7:4293 , piece_id=55K4BNAOG23667SL5RECDIIPCGNWVV33PTXRRV5NJX5DKMBM436Q, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:22.641685455 (516h0m22.51276849s) [upload=452760]: Next returned nil segment Dec 27 03:29:22.641706314 (516h0m22.512789359s) [upload=452760]: Waiting for error group managing segments... Dec 27 03:29:23.028099613 (516h0m22.899182658s) [noise=true , address=145.79.227.2:4113 , piece_id=7AIN6YI6RJZK6DLYVOZT5HRVAZDUEIWQJEYY6N5GJ7QS2YMXLBCA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.030891359 (516h0m22.901974394s) [noise=true , address=145.79.227.46:4603 , piece_id=EP7YIZBFN2VYWTDPKTDI2PCIWLJLDALQAHIZWCSRGR546ARYDCYA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.035562313 (516h0m22.906645388s) [noise=true , address=145.79.227.52:4447 , piece_id=4L3EUSG5TRSYNXQMHIC47O5P6CRGSZQ4NFE7NR2V6UADGF7PC7MQ, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.040130832 (516h0m22.911213867s) [noise=true , address=145.79.227.5:4221 , piece_id=B2QTSVBQDLEIEYT7XKHJAFGETP34SZDDYFHHWXGVSD7OU4ZDONSA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.053642391 (516h0m22.924725436s) [noise=true , address=145.79.227.38:4977 , piece_id=QBCPI2XH5Q7JFME77RTUXARSE3T2CF2R4GERNTLW2Y5DGYD275GA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.056420751 (516h0m22.927503786s) [noise=true , address=145.79.227.51:4410 , piece_id=5QJ2HOEEL2HXXQ6TBW43UIPZX3SSJDUCUE2JPE6CTQF7AVYIJYBQ, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.063262259 (516h0m22.934345294s) [noise=true , address=145.79.227.7:4293 , piece_id=55K4BNAOG23667SL5RECDIIPCGNWVV33PTXRRV5NJX5DKMBM436Q, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.223220185 (516h0m23.09430322s) [noise=true , address=145.79.227.13:4804 , piece_id=GF22Z64EUQPRBMXVLFDE5HXCNZQCCQZYN2EFHOE3GWEV6HOUUMUA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.275193321 (516h0m23.146276356s) [noise=true , address=77.90.31.13:4627 , piece_id=CS3FQ44IQAPZFINZ25A3NGFUGHUFNNSH3ZBS4JFUHONWX2NLXK5Q, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.276593236 (516h0m23.147676271s) [noise=true , address=45.91.116.4:4939 , piece_id=FYX6EVAOIELVDH3WVROJQLJF7QH3OKVNC2K55QAQCOSBLUPVQRDA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.320303388 (516h0m23.191386433s) [noise=true , address=45.91.116.11:4120 , piece_id=KXLM7LNJDFHJPJJPRAS6MJPUX2G52WBCLKM2GEOSIYXGAJAQNYZA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.357177533 (516h0m23.228260578s) [noise=true , address=77.90.31.10:4519 , piece_id=ZJNU5RHMFKZ2QYEHCZQQ56IVIMV5TEDCXEHNFBLJIFK5T4KGSKZQ, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:24.031054198 (516h0m23.902137233s) [noise=true , address=45.91.116.14:4191 , piece_id=YRFMS3A2X4OQQA3DQQQFCVFVZI6HLBSKHUQ4U7F5XXSN27M7O42Q, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:24.031075148 (516h0m23.902158193s) [seg_pos={0 0}, upload=452760]: Segment reached optimal threshold of 13 pieces. Dec 27 03:29:24.031300467 (516h0m23.902383512s) [noise=true , address=45.91.116.9:4262 , piece_id=7263XABPRAG4L7VYNRXMUP4ZW2CPWNFFFPPGRAXGU6C3ZULRRS3A, seg_pos={0 0}, upload=452760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 27 03:29:24.031307209 (516h0m23.902390245s) [noise=true , address=45.91.116.9:4262 , piece_id=7263XABPRAG4L7VYNRXMUP4ZW2CPWNFFFPPGRAXGU6C3ZULRRS3A, seg_pos={0 0}, upload=452760]: Long tail cancellation due to optimal threshold Dec 27 03:29:24.031520697 (516h0m23.902603742s) [noise=true , address=77.90.31.14:4661 , piece_id=CIFUDAKEQ5WZ54LXT4J2QOSS7PD4RVWB6B5LREDWLEYYC4NYIGZA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 03:29:24.031552177 (516h0m23.902635212s) [noise=true , address=77.90.31.14:4661 , piece_id=CIFUDAKEQ5WZ54LXT4J2QOSS7PD4RVWB6B5LREDWLEYYC4NYIGZA, seg_pos={0 0}, upload=452760]: Long tail cancellation due to optimal threshold Dec 27 03:29:24.031609305 (516h0m23.90269234s) [noise=true , address=45.91.116.10:4084 , piece_id=4LPIECHH5RFVH74J5BGN3THENYSGPHLAPNL3Q5WFFRMY3Q7UCXKA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 27 03:29:24.031617851 (516h0m23.902700886s) [noise=true , address=45.91.116.10:4084 , piece_id=4LPIECHH5RFVH74J5BGN3THENYSGPHLAPNL3Q5WFFRMY3Q7UCXKA, seg_pos={0 0}, upload=452760]: Long tail cancellation due to optimal threshold Dec 27 03:29:24.031636898 (516h0m23.902719943s) [seg_pos={0 0}, upload=452760]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:29:24.031768168 (516h0m23.902851213s) [upload=452760]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:29:24.031777816 (516h0m23.902860851s) [upload=452760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:29:24.313074291 (516h0m24.184157356s) [upload=452760]: Done uploading segments... Dec 27 03:29:24.313102956 (516h0m24.184186011s) [upload=452760]: Upload finished. err: <nil> Duration: 6.093911208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [X.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XXX] duration: 30.002600213s trace: https://jaeger.dev.storj.io/trace/68cfb97b8a21353f FAIL (2025-12-27 03:29:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: 7AIN6YI6RJZK6DLYVOZT5HRVAZDUEIWQJEYY6N5GJ7QS2YMXLBCA) piecestore close: write tcp 188.166.225.170:56336->145.79.227.2:4113: use of closed network connection; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZJNU5RHMFKZ2QYEHCZQQ56IVIMV5TEDCXEHNFBLJIFK5T4KGSKZQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: KXLM7LNJDFHJPJJPRAS6MJPUX2G52WBCLKM2GEOSIYXGAJAQNYZA) context deadline exceeded ======LOGS====== Dec 27 03:29:20.775216031 (516h0m20.646299066s) [upload=452760]: Starting upload Dec 27 03:29:20.775289100 (516h0m20.646372135s) [upload=452760]: Done starting upload Dec 27 03:29:20.775418817 (516h0m20.646501852s) [upload=452760]: Uploading segments... Dec 27 03:29:20.775540017 (516h0m20.646623052s) [upload=452760]: Got next segment. Inline: false Dec 27 03:29:20.775589933 (516h0m20.646672968s) [upload=452760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:29:20.775595163 (516h0m20.646678198s) [upload=452760]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:29:21.020329299 (516h0m20.891412344s) [seg_pos={0 0}, upload=452760]: Begin upload segment... Dec 27 03:29:21.020375938 (516h0m20.891458983s) [seg_pos={0 0}, upload=452760]: Done begin upload segment. Dec 27 03:29:21.020403511 (516h0m20.891486556s) [noise=true , address=145.79.227.51:4410 , piece_id=5QJ2HOEEL2HXXQ6TBW43UIPZX3SSJDUCUE2JPE6CTQF7AVYIJYBQ, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.020465458 (516h0m20.891548503s) [noise=true , address=145.79.227.5:4221 , piece_id=B2QTSVBQDLEIEYT7XKHJAFGETP34SZDDYFHHWXGVSD7OU4ZDONSA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.021059820 (516h0m20.892142865s) [noise=true , address=145.79.227.38:4977 , piece_id=QBCPI2XH5Q7JFME77RTUXARSE3T2CF2R4GERNTLW2Y5DGYD275GA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.021462236 (516h0m20.892545281s) [noise=true , address=145.79.227.2:4113 , piece_id=7AIN6YI6RJZK6DLYVOZT5HRVAZDUEIWQJEYY6N5GJ7QS2YMXLBCA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.021916080 (516h0m20.892999115s) [noise=true , address=77.90.31.10:4519 , piece_id=ZJNU5RHMFKZ2QYEHCZQQ56IVIMV5TEDCXEHNFBLJIFK5T4KGSKZQ, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.022296044 (516h0m20.893379089s) [noise=true , address=45.91.116.4:4939 , piece_id=FYX6EVAOIELVDH3WVROJQLJF7QH3OKVNC2K55QAQCOSBLUPVQRDA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.022777030 (516h0m20.893860065s) [noise=true , address=145.79.227.52:4447 , piece_id=4L3EUSG5TRSYNXQMHIC47O5P6CRGSZQ4NFE7NR2V6UADGF7PC7MQ, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.023593003 (516h0m20.894676038s) [noise=true , address=45.91.116.11:4120 , piece_id=KXLM7LNJDFHJPJJPRAS6MJPUX2G52WBCLKM2GEOSIYXGAJAQNYZA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.023657326 (516h0m20.894740371s) [noise=true , address=77.90.31.14:4661 , piece_id=CIFUDAKEQ5WZ54LXT4J2QOSS7PD4RVWB6B5LREDWLEYYC4NYIGZA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.024223995 (516h0m20.89530704s) [noise=true , address=145.79.227.46:4603 , piece_id=EP7YIZBFN2VYWTDPKTDI2PCIWLJLDALQAHIZWCSRGR546ARYDCYA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.024566888 (516h0m20.895649933s) [noise=true , address=45.91.116.9:4262 , piece_id=7263XABPRAG4L7VYNRXMUP4ZW2CPWNFFFPPGRAXGU6C3ZULRRS3A, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.025615274 (516h0m20.896698309s) [noise=true , address=45.91.116.14:4191 , piece_id=YRFMS3A2X4OQQA3DQQQFCVFVZI6HLBSKHUQ4U7F5XXSN27M7O42Q, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.026613816 (516h0m20.897696861s) [noise=true , address=45.91.116.10:4084 , piece_id=4LPIECHH5RFVH74J5BGN3THENYSGPHLAPNL3Q5WFFRMY3Q7UCXKA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.027385194 (516h0m20.898468229s) [noise=true , address=77.90.31.13:4627 , piece_id=CS3FQ44IQAPZFINZ25A3NGFUGHUFNNSH3ZBS4JFUHONWX2NLXK5Q, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.027723087 (516h0m20.898806122s) [noise=true , address=145.79.227.13:4804 , piece_id=GF22Z64EUQPRBMXVLFDE5HXCNZQCCQZYN2EFHOE3GWEV6HOUUMUA, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:21.028729223 (516h0m20.899812268s) [noise=true , address=145.79.227.7:4293 , piece_id=55K4BNAOG23667SL5RECDIIPCGNWVV33PTXRRV5NJX5DKMBM436Q, seg_pos={0 0}, upload=452760]: Uploading piece... Dec 27 03:29:22.641685455 (516h0m22.51276849s) [upload=452760]: Next returned nil segment Dec 27 03:29:22.641706314 (516h0m22.512789359s) [upload=452760]: Waiting for error group managing segments... Dec 27 03:29:23.028099613 (516h0m22.899182658s) [noise=true , address=145.79.227.2:4113 , piece_id=7AIN6YI6RJZK6DLYVOZT5HRVAZDUEIWQJEYY6N5GJ7QS2YMXLBCA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.030891359 (516h0m22.901974394s) [noise=true , address=145.79.227.46:4603 , piece_id=EP7YIZBFN2VYWTDPKTDI2PCIWLJLDALQAHIZWCSRGR546ARYDCYA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.035562313 (516h0m22.906645388s) [noise=true , address=145.79.227.52:4447 , piece_id=4L3EUSG5TRSYNXQMHIC47O5P6CRGSZQ4NFE7NR2V6UADGF7PC7MQ, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.040130832 (516h0m22.911213867s) [noise=true , address=145.79.227.5:4221 , piece_id=B2QTSVBQDLEIEYT7XKHJAFGETP34SZDDYFHHWXGVSD7OU4ZDONSA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.053642391 (516h0m22.924725436s) [noise=true , address=145.79.227.38:4977 , piece_id=QBCPI2XH5Q7JFME77RTUXARSE3T2CF2R4GERNTLW2Y5DGYD275GA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.056420751 (516h0m22.927503786s) [noise=true , address=145.79.227.51:4410 , piece_id=5QJ2HOEEL2HXXQ6TBW43UIPZX3SSJDUCUE2JPE6CTQF7AVYIJYBQ, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.063262259 (516h0m22.934345294s) [noise=true , address=145.79.227.7:4293 , piece_id=55K4BNAOG23667SL5RECDIIPCGNWVV33PTXRRV5NJX5DKMBM436Q, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.223220185 (516h0m23.09430322s) [noise=true , address=145.79.227.13:4804 , piece_id=GF22Z64EUQPRBMXVLFDE5HXCNZQCCQZYN2EFHOE3GWEV6HOUUMUA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.275193321 (516h0m23.146276356s) [noise=true , address=77.90.31.13:4627 , piece_id=CS3FQ44IQAPZFINZ25A3NGFUGHUFNNSH3ZBS4JFUHONWX2NLXK5Q, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.276593236 (516h0m23.147676271s) [noise=true , address=45.91.116.4:4939 , piece_id=FYX6EVAOIELVDH3WVROJQLJF7QH3OKVNC2K55QAQCOSBLUPVQRDA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.320303388 (516h0m23.191386433s) [noise=true , address=45.91.116.11:4120 , piece_id=KXLM7LNJDFHJPJJPRAS6MJPUX2G52WBCLKM2GEOSIYXGAJAQNYZA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:23.357177533 (516h0m23.228260578s) [noise=true , address=77.90.31.10:4519 , piece_id=ZJNU5RHMFKZ2QYEHCZQQ56IVIMV5TEDCXEHNFBLJIFK5T4KGSKZQ, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:24.031054198 (516h0m23.902137233s) [noise=true , address=45.91.116.14:4191 , piece_id=YRFMS3A2X4OQQA3DQQQFCVFVZI6HLBSKHUQ4U7F5XXSN27M7O42Q, seg_pos={0 0}, upload=452760]: Done uploading piece. err: <nil> Dec 27 03:29:24.031075148 (516h0m23.902158193s) [seg_pos={0 0}, upload=452760]: Segment reached optimal threshold of 13 pieces. Dec 27 03:29:24.031300467 (516h0m23.902383512s) [noise=true , address=45.91.116.9:4262 , piece_id=7263XABPRAG4L7VYNRXMUP4ZW2CPWNFFFPPGRAXGU6C3ZULRRS3A, seg_pos={0 0}, upload=452760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 27 03:29:24.031307209 (516h0m23.902390245s) [noise=true , address=45.91.116.9:4262 , piece_id=7263XABPRAG4L7VYNRXMUP4ZW2CPWNFFFPPGRAXGU6C3ZULRRS3A, seg_pos={0 0}, upload=452760]: Long tail cancellation due to optimal threshold Dec 27 03:29:24.031520697 (516h0m23.902603742s) [noise=true , address=77.90.31.14:4661 , piece_id=CIFUDAKEQ5WZ54LXT4J2QOSS7PD4RVWB6B5LREDWLEYYC4NYIGZA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 03:29:24.031552177 (516h0m23.902635212s) [noise=true , address=77.90.31.14:4661 , piece_id=CIFUDAKEQ5WZ54LXT4J2QOSS7PD4RVWB6B5LREDWLEYYC4NYIGZA, seg_pos={0 0}, upload=452760]: Long tail cancellation due to optimal threshold Dec 27 03:29:24.031609305 (516h0m23.90269234s) [noise=true , address=45.91.116.10:4084 , piece_id=4LPIECHH5RFVH74J5BGN3THENYSGPHLAPNL3Q5WFFRMY3Q7UCXKA, seg_pos={0 0}, upload=452760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 27 03:29:24.031617851 (516h0m23.902700886s) [noise=true , address=45.91.116.10:4084 , piece_id=4LPIECHH5RFVH74J5BGN3THENYSGPHLAPNL3Q5WFFRMY3Q7UCXKA, seg_pos={0 0}, upload=452760]: Long tail cancellation due to optimal threshold Dec 27 03:29:24.031636898 (516h0m23.902719943s) [seg_pos={0 0}, upload=452760]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:29:24.031768168 (516h0m23.902851213s) [upload=452760]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:29:24.031777816 (516h0m23.902860851s) [upload=452760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:29:24.313074291 (516h0m24.184157356s) [upload=452760]: Done uploading segments... Dec 27 03:29:24.313102956 (516h0m24.184186011s) [upload=452760]: Upload finished. err: <nil> Duration: 30.002615853s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.XX] duration: 30.002761216s trace: https://jaeger.dev.storj.io/trace/2d1ef8d42f60b6d3 FAIL (2025-12-27 03:29:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m50.418269612s) [upload=452754]: Done starting upload Dec 27 03:28:50.547265807 (515h59m50.418348842s) [upload=452754]: Uploading segments... Dec 27 03:28:50.547485286 (515h59m50.418568331s) [upload=452754]: Got next segment. Inline: false Dec 27 03:28:50.547514131 (515h59m50.418597166s) [upload=452754]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:28:50.547518399 (515h59m50.418601434s) [upload=452754]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:28:50.791112213 (515h59m50.662195248s) [seg_pos={0 0}, upload=452754]: Begin upload segment... Dec 27 03:28:50.791167718 (515h59m50.662250753s) [seg_pos={0 0}, upload=452754]: Done begin upload segment. Dec 27 03:28:50.791247230 (515h59m50.662330285s) [noise=true , address=145.79.227.17:4971 , piece_id=UC7NZW7UU4UAYOBZXY2PC5HEOGPQT4UHAJWCPY5JMKJJLMLATQ4Q, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.791268650 (515h59m50.662351695s) [noise=true , address=77.90.31.18:4808 , piece_id=4H7KL7GNJEQGNTP4ZG5EQ6G6QUC6I3NHVKDUXX5VBCFR7XTH4DRQ, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.792122917 (515h59m50.663205952s) [noise=true , address=145.79.227.40:4051 , piece_id=ZXD2PZZURQCQVRRN3GCWIQ3QPCO64RQ7B4JJRNRFX364OGBY7YUQ, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.792655812 (515h59m50.663738847s) [noise=true , address=145.79.227.19:4077 , piece_id=C4HZWHRJ56CKQ3LJV5PNJJ7Y353RFTAYIQMU5GZMPFZNAXJDQRMA, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.793087203 (515h59m50.664170238s) [noise=true , address=145.79.227.6:4259 , piece_id=QGE325QHWPDEXSHOUAP5ZMUCREBB7QAZCJ5WZJAH3U72XOC476KA, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.793106730 (515h59m50.664189776s) [noise=true , address=145.79.227.5:4221 , piece_id=SA745O5P4SRVPDM4JSBDPVKMCG3IQPCBZ5E53HM6QJ4ESMRHWSRA, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.793377637 (515h59m50.664460682s) [noise=true , address=145.79.227.11:4731 , piece_id=DADGBRXQCZ62YLKSSKGHW45IPL2NB7YZ5NVWMYSI4EZIRZGAA4FA, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.793881245 (515h59m50.66496429s) [noise=true , address=45.91.116.6:4976 , piece_id=NO6DY3LXQPZMNEAJ3PMG2WD76E2BFMO3FDZE3RHIDQFFLHZ6IBPQ, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.794026752 (515h59m50.665109787s) [noise=true , address=45.91.116.11:4119 , piece_id=4L7VIUP7WV7BABHJ64GJC4FPE26ENIP6D3WUZX7JCEHAJIDNARAQ, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.794285966 (515h59m50.665369011s) [noise=true , address=145.79.227.36:4517 , piece_id=7OY5IJOXM3ZWA3PMEPQZ3QKBEWRX2BKXT7QYMSCS2DMZ2HDANAZQ, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.794780998 (515h59m50.665864034s) [noise=true , address=45.91.116.7:4010 , piece_id=T25CJ27MQOYSIO7OHJPBXY6O3BFWV4NVJ3JS3CIEXELUZOA25TQA, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.794930864 (515h59m50.666013899s) [noise=true , address=45.91.116.9:4264 , piece_id=BAPM7EVTR2HQ2LNUSTUCE4PHVKT64YNENM2QFIJULJ5JZX4XHEZA, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.795244200 (515h59m50.666327235s) [noise=true , address=77.90.31.19:4841 , piece_id=CWZJWCQAVIHLEC2YUBCJKNJTOYOQRCE6DKUSZ224XSZB6WDZU4OQ, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.795398254 (515h59m50.666481299s) [noise=true , address=145.79.227.16:4911 , piece_id=EDSGQCUWRHLRZGZ2IABQ35KYPOOYYFS3U4K522MVDDKLFRILRJAQ, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.795991474 (515h59m50.667074509s) [noise=true , address=145.79.227.42:4122 , piece_id=WV4XOVUADDNA4QJBCTHIA44VO42AKRKOOERTPXLCANUNRL5OQ2GA, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:50.797887864 (515h59m50.668970899s) [noise=true , address=145.79.227.41:4085 , piece_id=EF6NKXT62NC5CLWAWZ4OF7MUAAS5HYHN67XCWG234SXJCW3ZNDSQ, seg_pos={0 0}, upload=452754]: Uploading piece... Dec 27 03:28:52.499427915 (515h59m52.37051095s) [upload=452754]: Next returned nil segment Dec 27 03:28:52.499442303 (515h59m52.370525338s) [upload=452754]: Waiting for error group managing segments... Dec 27 03:28:52.781390138 (515h59m52.652473173s) [noise=true , address=145.79.227.11:4731 , piece_id=DADGBRXQCZ62YLKSSKGHW45IPL2NB7YZ5NVWMYSI4EZIRZGAA4FA, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:52.784529336 (515h59m52.655612381s) [noise=true , address=145.79.227.36:4517 , piece_id=7OY5IJOXM3ZWA3PMEPQZ3QKBEWRX2BKXT7QYMSCS2DMZ2HDANAZQ, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:52.791563149 (515h59m52.662646184s) [noise=true , address=145.79.227.6:4259 , piece_id=QGE325QHWPDEXSHOUAP5ZMUCREBB7QAZCJ5WZJAH3U72XOC476KA, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:52.796205770 (515h59m52.667288815s) [noise=true , address=145.79.227.17:4971 , piece_id=UC7NZW7UU4UAYOBZXY2PC5HEOGPQT4UHAJWCPY5JMKJJLMLATQ4Q, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:52.807192490 (515h59m52.678275535s) [noise=true , address=145.79.227.16:4911 , piece_id=EDSGQCUWRHLRZGZ2IABQ35KYPOOYYFS3U4K522MVDDKLFRILRJAQ, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:52.813915954 (515h59m52.684998989s) [noise=true , address=145.79.227.5:4221 , piece_id=SA745O5P4SRVPDM4JSBDPVKMCG3IQPCBZ5E53HM6QJ4ESMRHWSRA, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:52.830796567 (515h59m52.701879603s) [noise=true , address=145.79.227.41:4085 , piece_id=EF6NKXT62NC5CLWAWZ4OF7MUAAS5HYHN67XCWG234SXJCW3ZNDSQ, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:52.986314639 (515h59m52.857397694s) [noise=true , address=77.90.31.18:4808 , piece_id=4H7KL7GNJEQGNTP4ZG5EQ6G6QUC6I3NHVKDUXX5VBCFR7XTH4DRQ, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:53.066035381 (515h59m52.937118426s) [noise=true , address=45.91.116.11:4119 , piece_id=4L7VIUP7WV7BABHJ64GJC4FPE26ENIP6D3WUZX7JCEHAJIDNARAQ, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:53.108074942 (515h59m52.979157987s) [noise=true , address=77.90.31.19:4841 , piece_id=CWZJWCQAVIHLEC2YUBCJKNJTOYOQRCE6DKUSZ224XSZB6WDZU4OQ, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:53.144157339 (515h59m53.015240384s) [noise=true , address=45.91.116.6:4976 , piece_id=NO6DY3LXQPZMNEAJ3PMG2WD76E2BFMO3FDZE3RHIDQFFLHZ6IBPQ, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:53.215066091 (515h59m53.086149126s) [noise=true , address=45.91.116.9:4264 , piece_id=BAPM7EVTR2HQ2LNUSTUCE4PHVKT64YNENM2QFIJULJ5JZX4XHEZA, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:53.309147728 (515h59m53.180230773s) [noise=true , address=145.79.227.40:4051 , piece_id=ZXD2PZZURQCQVRRN3GCWIQ3QPCO64RQ7B4JJRNRFX364OGBY7YUQ, seg_pos={0 0}, upload=452754]: Done uploading piece. err: <nil> Dec 27 03:28:53.309172816 (515h59m53.180255861s) [seg_pos={0 0}, upload=452754]: Segment reached optimal threshold of 13 pieces. Dec 27 03:28:53.309645886 (515h59m53.180728932s) [noise=true , address=145.79.227.19:4077 , piece_id=C4HZWHRJ56CKQ3LJV5PNJJ7Y353RFTAYIQMU5GZMPFZNAXJDQRMA, seg_pos={0 0}, upload=452754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:50462->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:50462->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50462->145.79.227.19:4077: use of closed network connection Dec 27 03:28:53.309655656 (515h59m53.180738691s) [noise=true , address=145.79.227.19:4077 , piece_id=C4HZWHRJ56CKQ3LJV5PNJJ7Y353RFTAYIQMU5GZMPFZNAXJDQRMA, seg_pos={0 0}, upload=452754]: Long tail cancellation due to optimal threshold Dec 27 03:28:53.309789159 (515h59m53.180872204s) [noise=true , address=145.79.227.42:4122 , piece_id=WV4XOVUADDNA4QJBCTHIA44VO42AKRKOOERTPXLCANUNRL5OQ2GA, seg_pos={0 0}, upload=452754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 27 03:28:53.309903658 (515h59m53.180986703s) [noise=true , address=145.79.227.42:4122 , piece_id=WV4XOVUADDNA4QJBCTHIA44VO42AKRKOOERTPXLCANUNRL5OQ2GA, seg_pos={0 0}, upload=452754]: Long tail cancellation due to optimal threshold Dec 27 03:28:53.310183441 (515h59m53.181266486s) [noise=true , address=45.91.116.7:4010 , piece_id=T25CJ27MQOYSIO7OHJPBXY6O3BFWV4NVJ3JS3CIEXELUZOA25TQA, seg_pos={0 0}, upload=452754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:35746->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 03:28:53.310194341 (515h59m53.181277406s) [noise=true , address=45.91.116.7:4010 , piece_id=T25CJ27MQOYSIO7OHJPBXY6O3BFWV4NVJ3JS3CIEXELUZOA25TQA, seg_pos={0 0}, upload=452754]: Long tail cancellation due to optimal threshold Dec 27 03:28:53.310219910 (515h59m53.181302955s) [seg_pos={0 0}, upload=452754]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:28:53.310334368 (515h59m53.181417413s) [upload=452754]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:28:53.310341752 (515h59m53.181424797s) [upload=452754]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:28:53.594358528 (515h59m53.465441573s) [upload=452754]: Done uploading segments... Dec 27 03:28:53.594502121 (515h59m53.465585166s) [upload=452754]: Upload finished. err: <nil> Duration: 30.002781774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.X] duration: 30.003020386s trace: https://jaeger.dev.storj.io/trace/50a9ea24e7456c41 FAIL (2025-12-27 03:28:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (515h59m13.606187512s) [upload=452743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:28:13.989234664 (515h59m13.86031771s) [seg_pos={0 0}, upload=452743]: Begin upload segment... Dec 27 03:28:13.989283838 (515h59m13.860366883s) [seg_pos={0 0}, upload=452743]: Done begin upload segment. Dec 27 03:28:13.989315147 (515h59m13.860398192s) [noise=true , address=145.79.227.9:4365 , piece_id=JYCN5TKZQ5U6PX2TKRBZK4FCJWPRFFZTD3CB6OXYGIDZJOKVZBWQ, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.989459462 (515h59m13.860542497s) [noise=true , address=145.79.227.6:4259 , piece_id=UHZBQIA7MINNGHLHWONC2E4J5CGKBBC6QIVE73TSC3N34N62JTCQ, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.989918616 (515h59m13.861001651s) [noise=true , address=145.79.227.3:4149 , piece_id=ATZUVJHSPE3V4ADKTMWACEO5V4GJXHPEBP5S2ZT6SUYN5HUSCZDA, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.990289112 (515h59m13.861372157s) [noise=true , address=145.79.227.38:4978 , piece_id=UJJWJWI5FXDZAO5WIJ2NYLWMWELDY46UVC33HJUGPE35GLSQDWGQ, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.990747144 (515h59m13.861830189s) [noise=true , address=145.79.227.37:4565 , piece_id=CSZKE3KJBFW22CW7LHH3RLTOAWFPHZG4ZBD6OIVJ3454S2ANNCKA, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.991068536 (515h59m13.862151581s) [noise=true , address=145.79.227.16:4911 , piece_id=EUJ3757K2TH3LQWKJHXKNDDGZNGZWOCWLYE6IP7MCJDN2RWKSVGA, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.991381732 (515h59m13.862464767s) [noise=true , address=45.91.116.6:4973 , piece_id=3DYREHQ42YLHPYOTPJ57WWZX43YO2P3RFONV4UF5JAVLLN66AHJQ, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.991635556 (515h59m13.862718591s) [noise=true , address=145.79.227.48:4301 , piece_id=X66BB2EAWLMVLP7V55BFLRS7BE6LHGVYBU3FBG2AZEITKJH5J73Q, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.991884200 (515h59m13.862967235s) [noise=true , address=145.79.227.19:4077 , piece_id=25BPKT7M37EZV234GAWE4ULFOB7NGHHHJKAHYMOEHZQTSHBM5LVA, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.993392431 (515h59m13.864475466s) [noise=true , address=77.90.31.17:4770 , piece_id=PY3GAM66JDYQ72IIGOLY6C756RJC56WJEJVG5HH5NA4A5YE7L55A, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.993713212 (515h59m13.864796247s) [noise=true , address=145.79.227.30:4762 , piece_id=CZVOS4TN74OYVRULRA3A4WAQCGKOP4UVWKUGIIUUBO2TY4N47EEA, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.994232781 (515h59m13.865315816s) [noise=true , address=145.79.227.47:4267 , piece_id=4J6UPKTV2ZEZWMEDZZJVVAF4GGAK6YYX2O3R6EXVCOKAPVVOFWFA, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.994642972 (515h59m13.865726008s) [noise=true , address=145.79.227.5:4223 , piece_id=OOUQPCLOK7F34JGXMHNXD3FI4CWWXPS47PHN4B4ZS5JFQPFT6PWQ, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.994964474 (515h59m13.866047509s) [noise=true , address=145.79.227.46:4602 , piece_id=PXVXTX5LWAFOV72L6RHEQXL6EQIGOXXXBO7A2DERR6D2G54NIF7Q, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.995383071 (515h59m13.866466107s) [noise=true , address=145.79.227.49:4339 , piece_id=6HFXS5PHBGOROIN5PPXWYG6VTOJKSFVU5GEYDKFVMVJUIWGGHH5Q, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:13.996042487 (515h59m13.867125522s) [noise=true , address=77.90.31.10:4518 , piece_id=Q5XM2MESZ64FCX3AJJVPGVUF4D4DR2MK2JVRK3D5QGIFUSH77TJQ, seg_pos={0 0}, upload=452743]: Uploading piece... Dec 27 03:28:15.610153087 (515h59m15.481236132s) [upload=452743]: Next returned nil segment Dec 27 03:28:15.610189587 (515h59m15.481272622s) [upload=452743]: Waiting for error group managing segments... Dec 27 03:28:15.993418941 (515h59m15.864501976s) [noise=true , address=145.79.227.3:4149 , piece_id=ATZUVJHSPE3V4ADKTMWACEO5V4GJXHPEBP5S2ZT6SUYN5HUSCZDA, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:15.998605577 (515h59m15.869688622s) [noise=true , address=145.79.227.38:4978 , piece_id=UJJWJWI5FXDZAO5WIJ2NYLWMWELDY46UVC33HJUGPE35GLSQDWGQ, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:16.001342017 (515h59m15.872425062s) [noise=true , address=145.79.227.49:4339 , piece_id=6HFXS5PHBGOROIN5PPXWYG6VTOJKSFVU5GEYDKFVMVJUIWGGHH5Q, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:16.006181903 (515h59m15.877264938s) [noise=true , address=145.79.227.9:4365 , piece_id=JYCN5TKZQ5U6PX2TKRBZK4FCJWPRFFZTD3CB6OXYGIDZJOKVZBWQ, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:16.008461013 (515h59m15.879544048s) [noise=true , address=145.79.227.16:4911 , piece_id=EUJ3757K2TH3LQWKJHXKNDDGZNGZWOCWLYE6IP7MCJDN2RWKSVGA, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:16.011706122 (515h59m15.882789167s) [noise=true , address=145.79.227.48:4301 , piece_id=X66BB2EAWLMVLP7V55BFLRS7BE6LHGVYBU3FBG2AZEITKJH5J73Q, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:16.012589545 (515h59m15.8836726s) [noise=true , address=145.79.227.37:4565 , piece_id=CSZKE3KJBFW22CW7LHH3RLTOAWFPHZG4ZBD6OIVJ3454S2ANNCKA, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:16.023017561 (515h59m15.894100606s) [noise=true , address=145.79.227.47:4267 , piece_id=4J6UPKTV2ZEZWMEDZZJVVAF4GGAK6YYX2O3R6EXVCOKAPVVOFWFA, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:16.028016349 (515h59m15.899099384s) [noise=true , address=145.79.227.46:4602 , piece_id=PXVXTX5LWAFOV72L6RHEQXL6EQIGOXXXBO7A2DERR6D2G54NIF7Q, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:16.299174440 (515h59m16.170257486s) [noise=true , address=77.90.31.17:4770 , piece_id=PY3GAM66JDYQ72IIGOLY6C756RJC56WJEJVG5HH5NA4A5YE7L55A, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:16.345410802 (515h59m16.216493847s) [noise=true , address=45.91.116.6:4973 , piece_id=3DYREHQ42YLHPYOTPJ57WWZX43YO2P3RFONV4UF5JAVLLN66AHJQ, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:16.603328571 (515h59m16.474411616s) [noise=true , address=77.90.31.10:4518 , piece_id=Q5XM2MESZ64FCX3AJJVPGVUF4D4DR2MK2JVRK3D5QGIFUSH77TJQ, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:17.341284344 (515h59m17.212367389s) [noise=true , address=145.79.227.30:4762 , piece_id=CZVOS4TN74OYVRULRA3A4WAQCGKOP4UVWKUGIIUUBO2TY4N47EEA, seg_pos={0 0}, upload=452743]: Done uploading piece. err: <nil> Dec 27 03:28:17.341309563 (515h59m17.212392598s) [seg_pos={0 0}, upload=452743]: Segment reached optimal threshold of 13 pieces. Dec 27 03:28:17.341597080 (515h59m17.212680125s) [noise=true , address=145.79.227.19:4077 , piece_id=25BPKT7M37EZV234GAWE4ULFOB7NGHHHJKAHYMOEHZQTSHBM5LVA, seg_pos={0 0}, upload=452743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:34608->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:34608->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34608->145.79.227.19:4077: use of closed network connection Dec 27 03:28:17.341617940 (515h59m17.212700995s) [noise=true , address=145.79.227.19:4077 , piece_id=25BPKT7M37EZV234GAWE4ULFOB7NGHHHJKAHYMOEHZQTSHBM5LVA, seg_pos={0 0}, upload=452743]: Long tail cancellation due to optimal threshold Dec 27 03:28:17.341646704 (515h59m17.212729739s) [noise=true , address=145.79.227.5:4223 , piece_id=OOUQPCLOK7F34JGXMHNXD3FI4CWWXPS47PHN4B4ZS5JFQPFT6PWQ, seg_pos={0 0}, upload=452743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:40944->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:40944->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40944->145.79.227.5:4223: use of closed network connection Dec 27 03:28:17.341658126 (515h59m17.212741171s) [noise=true , address=145.79.227.5:4223 , piece_id=OOUQPCLOK7F34JGXMHNXD3FI4CWWXPS47PHN4B4ZS5JFQPFT6PWQ, seg_pos={0 0}, upload=452743]: Long tail cancellation due to optimal threshold Dec 27 03:28:17.341980019 (515h59m17.213063074s) [noise=true , address=145.79.227.6:4259 , piece_id=UHZBQIA7MINNGHLHWONC2E4J5CGKBBC6QIVE73TSC3N34N62JTCQ, seg_pos={0 0}, upload=452743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:40574->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.6:4259: use of closed network connection Dec 27 03:28:17.341991060 (515h59m17.213074095s) [noise=true , address=145.79.227.6:4259 , piece_id=UHZBQIA7MINNGHLHWONC2E4J5CGKBBC6QIVE73TSC3N34N62JTCQ, seg_pos={0 0}, upload=452743]: Long tail cancellation due to optimal threshold Dec 27 03:28:17.342035345 (515h59m17.21311839s) [seg_pos={0 0}, upload=452743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:28:17.342126307 (515h59m17.213209352s) [upload=452743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:28:17.342133561 (515h59m17.213216606s) [upload=452743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:28:17.645968824 (515h59m17.51705187s) [upload=452743]: Done uploading segments... Dec 27 03:28:17.645988252 (515h59m17.517071287s) [upload=452743]: Upload finished. err: <nil> Duration:* 30.003039922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X.] duration: 5.704941102s trace: https://jaeger.dev.storj.io/trace/4678e6bb3be89ab8 FAIL (2025-12-27 03:28:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 03:27:28.310952914 (515h58m28.182035949s) [upload=452732]: Starting upload Dec 27 03:27:28.311012648 (515h58m28.182095683s) [upload=452732]: Done starting upload Dec 27 03:27:28.311120003 (515h58m28.182203048s) [upload=452732]: Uploading segments... Dec 27 03:27:28.311213311 (515h58m28.182296356s) [upload=452732]: Got next segment. Inline: false Dec 27 03:27:28.311274027 (515h58m28.182357062s) [upload=452732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:27:28.311279687 (515h58m28.182362722s) [upload=452732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:27:28.552764364 (515h58m28.423847399s) [seg_pos={0 0}, upload=452732]: Begin upload segment... Dec 27 03:27:28.552808859 (515h58m28.423891894s) [seg_pos={0 0}, upload=452732]: Done begin upload segment. Dec 27 03:27:28.552839927 (515h58m28.423922993s) [noise=true , address=145.79.227.8:4330 , piece_id=OEEEMUWMDQLCLRT7P3CPAD3RXQD2WQD256OE3RKSE7UPKLHXJTDQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.552894913 (515h58m28.423977948s) [noise=true , address=145.79.227.30:4761 , piece_id=FEZCPQFHCZDECIGPDLUB667YGROCXJ4MS67CLO243LQZGILEPHWQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.553303260 (515h58m28.424386295s) [noise=true , address=145.79.227.16:4912 , piece_id=XUF5MZBKUWL744XPDYPCXWWVT7ITUVCDCEQ2S7RNM2S4ZX3LKODA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.553684937 (515h58m28.424767982s) [noise=true , address=45.91.116.12:4226 , piece_id=2PVJKD2URAJ4CEYGQTMPY6DO6XU3V4OLDFYLUSNRK2DLBEAT5YVQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.554038630 (515h58m28.425121676s) [noise=true , address=145.79.227.35:4481 , piece_id=DIYCBOGZ36F5QIGGVIYIEDSUZEZDXNK3W6ZBB3SLRYG5S6GOMPIA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.554320457 (515h58m28.425403492s) [noise=true , address=145.79.227.36:4517 , piece_id=CQGAT3W4HOK6LY4QGHF5JSISWRBI57EMSB2PPU4DXRMOG3OKF3PQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.554326238 (515h58m28.425409273s) [noise=true , address=45.91.116.10:4083 , piece_id=ONSNWUB3WA4JOEOUTUGQOD54YVE4DFNKA4VH7GPDGBHYFUEGIJ4A, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.554776215 (515h58m28.42585925s) [noise=true , address=145.79.227.39:4015 , piece_id=V4L2NIGEBHAXNY2LL62K25GOLG4RU723B2YN7ZMLQXRVQHGKHANA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.554929827 (515h58m28.426012862s) [noise=true , address=145.79.227.52:4445 , piece_id=NLTAUOSCJCJAZD2ZEHHOGTJIWXGDJKVPOUDB4CLIPQV77URSPJOA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.555137773 (515h58m28.426220808s) [noise=true , address=145.79.227.47:4265 , piece_id=G3UNZA6XFSSLHWYS5SKZ7P3IM5GQBZACNIEJJZTYTG5UM5IRZ66A, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.555465718 (515h58m28.426548753s) [noise=true , address=45.91.116.9:4261 , piece_id=QESJQZSRVGCTZZDKKEDHTPBFHCIKI2WOMIFQTAFOIH3HDP7U7JOA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.555532756 (515h58m28.426615791s) [noise=true , address=145.79.227.6:4259 , piece_id=BZSC5D5DOJJPRVB7INXOIEH7ZAXGNUDU5CWIB3EJLHTSEWQ6O6HA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.555849629 (515h58m28.426932674s) [noise=true , address=45.91.116.13:4155 , piece_id=P6XKCDK4X7I3XDD7PU5B4QJUWQ7VL6WXB77LZVU6DINTZILHBSUA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.556345303 (515h58m28.427428338s) [noise=true , address=77.90.31.11:4554 , piece_id=OEBFHSK6V5V5IMDNX3AKQPWVFCGOBDOEFJI37VHRSCVBRJUTW5CQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.556791472 (515h58m28.427874507s) [noise=true , address=45.91.116.11:4120 , piece_id=64RL6MY43NZ6IKR32HXD3VAX434T6I7UDH72QFRQJLXT25QMJHYA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.557186745 (515h58m28.42826979s) [noise=true , address=145.79.227.11:4729 , piece_id=E6CVUNGOU7M7LSSNYBSP526FU6VXFWZUM47XJZD4E7MZON65CTGQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:30.264236767 (515h58m30.135319812s) [upload=452732]: Next returned nil segment Dec 27 03:27:30.264249210 (515h58m30.135332245s) [upload=452732]: Waiting for error group managing segments... Dec 27 03:27:30.556617729 (515h58m30.427700774s) [noise=true , address=145.79.227.39:4015 , piece_id=V4L2NIGEBHAXNY2LL62K25GOLG4RU723B2YN7ZMLQXRVQHGKHANA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.561429773 (515h58m30.432512808s) [noise=true , address=145.79.227.11:4729 , piece_id=E6CVUNGOU7M7LSSNYBSP526FU6VXFWZUM47XJZD4E7MZON65CTGQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.563700466 (515h58m30.434783502s) [noise=true , address=145.79.227.16:4912 , piece_id=XUF5MZBKUWL744XPDYPCXWWVT7ITUVCDCEQ2S7RNM2S4ZX3LKODA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.566802163 (515h58m30.437885198s) [noise=true , address=145.79.227.6:4259 , piece_id=BZSC5D5DOJJPRVB7INXOIEH7ZAXGNUDU5CWIB3EJLHTSEWQ6O6HA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.579720010 (515h58m30.450803055s) [noise=true , address=145.79.227.52:4445 , piece_id=NLTAUOSCJCJAZD2ZEHHOGTJIWXGDJKVPOUDB4CLIPQV77URSPJOA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.583646848 (515h58m30.454729893s) [noise=true , address=145.79.227.36:4517 , piece_id=CQGAT3W4HOK6LY4QGHF5JSISWRBI57EMSB2PPU4DXRMOG3OKF3PQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.587119731 (515h58m30.458202776s) [noise=true , address=145.79.227.47:4265 , piece_id=G3UNZA6XFSSLHWYS5SKZ7P3IM5GQBZACNIEJJZTYTG5UM5IRZ66A, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.621416848 (515h58m30.492499894s) [noise=true , address=145.79.227.8:4330 , piece_id=OEEEMUWMDQLCLRT7P3CPAD3RXQD2WQD256OE3RKSE7UPKLHXJTDQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.792748072 (515h58m30.663831107s) [noise=true , address=45.91.116.10:4083 , piece_id=ONSNWUB3WA4JOEOUTUGQOD54YVE4DFNKA4VH7GPDGBHYFUEGIJ4A, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.819862232 (515h58m30.690945277s) [noise=true , address=45.91.116.11:4120 , piece_id=64RL6MY43NZ6IKR32HXD3VAX434T6I7UDH72QFRQJLXT25QMJHYA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.856879379 (515h58m30.727962424s) [noise=true , address=45.91.116.9:4261 , piece_id=QESJQZSRVGCTZZDKKEDHTPBFHCIKI2WOMIFQTAFOIH3HDP7U7JOA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.857410149 (515h58m30.728493184s) [noise=true , address=45.91.116.12:4226 , piece_id=2PVJKD2URAJ4CEYGQTMPY6DO6XU3V4OLDFYLUSNRK2DLBEAT5YVQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.860047942 (515h58m30.731130977s) [noise=true , address=77.90.31.11:4554 , piece_id=OEBFHSK6V5V5IMDNX3AKQPWVFCGOBDOEFJI37VHRSCVBRJUTW5CQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.860061588 (515h58m30.731144633s) [seg_pos={0 0}, upload=452732]: Segment reached optimal threshold of 13 pieces. Dec 27 03:27:30.860271858 (515h58m30.731354893s) [noise=true , address=145.79.227.30:4761 , piece_id=FEZCPQFHCZDECIGPDLUB667YGROCXJ4MS67CLO243LQZGILEPHWQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 03:27:30.860282198 (515h58m30.731365234s) [noise=true , address=145.79.227.30:4761 , piece_id=FEZCPQFHCZDECIGPDLUB667YGROCXJ4MS67CLO243LQZGILEPHWQ, seg_pos={0 0}, upload=452732]: Long tail cancellation due to optimal threshold Dec 27 03:27:30.860398540 (515h58m30.731481585s) [noise=true , address=145.79.227.35:4481 , piece_id=DIYCBOGZ36F5QIGGVIYIEDSUZEZDXNK3W6ZBB3SLRYG5S6GOMPIA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 27 03:27:30.860412918 (515h58m30.731495953s) [noise=true , address=145.79.227.35:4481 , piece_id=DIYCBOGZ36F5QIGGVIYIEDSUZEZDXNK3W6ZBB3SLRYG5S6GOMPIA, seg_pos={0 0}, upload=452732]: Long tail cancellation due to optimal threshold Dec 27 03:27:30.860500885 (515h58m30.73158392s) [noise=true , address=45.91.116.13:4155 , piece_id=P6XKCDK4X7I3XDD7PU5B4QJUWQ7VL6WXB77LZVU6DINTZILHBSUA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 03:27:30.860506896 (515h58m30.731589941s) [noise=true , address=45.91.116.13:4155 , piece_id=P6XKCDK4X7I3XDD7PU5B4QJUWQ7VL6WXB77LZVU6DINTZILHBSUA, seg_pos={0 0}, upload=452732]: Long tail cancellation due to optimal threshold Dec 27 03:27:30.860522697 (515h58m30.731605732s) [seg_pos={0 0}, upload=452732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:27:30.860586447 (515h58m30.731669492s) [upload=452732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:27:30.860591067 (515h58m30.731674102s) [upload=452732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:27:31.155005780 (515h58m31.026088815s) [upload=452732]: Done uploading segments... Dec 27 03:27:31.155026508 (515h58m31.026109554s) [upload=452732]: Upload finished. err: <nil> Duration: 5.704982301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [X.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.X] duration: 30.003950944s trace: https://jaeger.dev.storj.io/trace/4678e6bb3be89ab8 FAIL (2025-12-27 03:27:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 03:27:28.310952914 (515h58m28.182035949s) [upload=452732]: Starting upload Dec 27 03:27:28.311012648 (515h58m28.182095683s) [upload=452732]: Done starting upload Dec 27 03:27:28.311120003 (515h58m28.182203048s) [upload=452732]: Uploading segments... Dec 27 03:27:28.311213311 (515h58m28.182296356s) [upload=452732]: Got next segment. Inline: false Dec 27 03:27:28.311274027 (515h58m28.182357062s) [upload=452732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:27:28.311279687 (515h58m28.182362722s) [upload=452732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:27:28.552764364 (515h58m28.423847399s) [seg_pos={0 0}, upload=452732]: Begin upload segment... Dec 27 03:27:28.552808859 (515h58m28.423891894s) [seg_pos={0 0}, upload=452732]: Done begin upload segment. Dec 27 03:27:28.552839927 (515h58m28.423922993s) [noise=true , address=145.79.227.8:4330 , piece_id=OEEEMUWMDQLCLRT7P3CPAD3RXQD2WQD256OE3RKSE7UPKLHXJTDQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.552894913 (515h58m28.423977948s) [noise=true , address=145.79.227.30:4761 , piece_id=FEZCPQFHCZDECIGPDLUB667YGROCXJ4MS67CLO243LQZGILEPHWQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.553303260 (515h58m28.424386295s) [noise=true , address=145.79.227.16:4912 , piece_id=XUF5MZBKUWL744XPDYPCXWWVT7ITUVCDCEQ2S7RNM2S4ZX3LKODA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.553684937 (515h58m28.424767982s) [noise=true , address=45.91.116.12:4226 , piece_id=2PVJKD2URAJ4CEYGQTMPY6DO6XU3V4OLDFYLUSNRK2DLBEAT5YVQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.554038630 (515h58m28.425121676s) [noise=true , address=145.79.227.35:4481 , piece_id=DIYCBOGZ36F5QIGGVIYIEDSUZEZDXNK3W6ZBB3SLRYG5S6GOMPIA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.554320457 (515h58m28.425403492s) [noise=true , address=145.79.227.36:4517 , piece_id=CQGAT3W4HOK6LY4QGHF5JSISWRBI57EMSB2PPU4DXRMOG3OKF3PQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.554326238 (515h58m28.425409273s) [noise=true , address=45.91.116.10:4083 , piece_id=ONSNWUB3WA4JOEOUTUGQOD54YVE4DFNKA4VH7GPDGBHYFUEGIJ4A, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.554776215 (515h58m28.42585925s) [noise=true , address=145.79.227.39:4015 , piece_id=V4L2NIGEBHAXNY2LL62K25GOLG4RU723B2YN7ZMLQXRVQHGKHANA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.554929827 (515h58m28.426012862s) [noise=true , address=145.79.227.52:4445 , piece_id=NLTAUOSCJCJAZD2ZEHHOGTJIWXGDJKVPOUDB4CLIPQV77URSPJOA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.555137773 (515h58m28.426220808s) [noise=true , address=145.79.227.47:4265 , piece_id=G3UNZA6XFSSLHWYS5SKZ7P3IM5GQBZACNIEJJZTYTG5UM5IRZ66A, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.555465718 (515h58m28.426548753s) [noise=true , address=45.91.116.9:4261 , piece_id=QESJQZSRVGCTZZDKKEDHTPBFHCIKI2WOMIFQTAFOIH3HDP7U7JOA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.555532756 (515h58m28.426615791s) [noise=true , address=145.79.227.6:4259 , piece_id=BZSC5D5DOJJPRVB7INXOIEH7ZAXGNUDU5CWIB3EJLHTSEWQ6O6HA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.555849629 (515h58m28.426932674s) [noise=true , address=45.91.116.13:4155 , piece_id=P6XKCDK4X7I3XDD7PU5B4QJUWQ7VL6WXB77LZVU6DINTZILHBSUA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.556345303 (515h58m28.427428338s) [noise=true , address=77.90.31.11:4554 , piece_id=OEBFHSK6V5V5IMDNX3AKQPWVFCGOBDOEFJI37VHRSCVBRJUTW5CQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.556791472 (515h58m28.427874507s) [noise=true , address=45.91.116.11:4120 , piece_id=64RL6MY43NZ6IKR32HXD3VAX434T6I7UDH72QFRQJLXT25QMJHYA, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:28.557186745 (515h58m28.42826979s) [noise=true , address=145.79.227.11:4729 , piece_id=E6CVUNGOU7M7LSSNYBSP526FU6VXFWZUM47XJZD4E7MZON65CTGQ, seg_pos={0 0}, upload=452732]: Uploading piece... Dec 27 03:27:30.264236767 (515h58m30.135319812s) [upload=452732]: Next returned nil segment Dec 27 03:27:30.264249210 (515h58m30.135332245s) [upload=452732]: Waiting for error group managing segments... Dec 27 03:27:30.556617729 (515h58m30.427700774s) [noise=true , address=145.79.227.39:4015 , piece_id=V4L2NIGEBHAXNY2LL62K25GOLG4RU723B2YN7ZMLQXRVQHGKHANA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.561429773 (515h58m30.432512808s) [noise=true , address=145.79.227.11:4729 , piece_id=E6CVUNGOU7M7LSSNYBSP526FU6VXFWZUM47XJZD4E7MZON65CTGQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.563700466 (515h58m30.434783502s) [noise=true , address=145.79.227.16:4912 , piece_id=XUF5MZBKUWL744XPDYPCXWWVT7ITUVCDCEQ2S7RNM2S4ZX3LKODA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.566802163 (515h58m30.437885198s) [noise=true , address=145.79.227.6:4259 , piece_id=BZSC5D5DOJJPRVB7INXOIEH7ZAXGNUDU5CWIB3EJLHTSEWQ6O6HA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.579720010 (515h58m30.450803055s) [noise=true , address=145.79.227.52:4445 , piece_id=NLTAUOSCJCJAZD2ZEHHOGTJIWXGDJKVPOUDB4CLIPQV77URSPJOA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.583646848 (515h58m30.454729893s) [noise=true , address=145.79.227.36:4517 , piece_id=CQGAT3W4HOK6LY4QGHF5JSISWRBI57EMSB2PPU4DXRMOG3OKF3PQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.587119731 (515h58m30.458202776s) [noise=true , address=145.79.227.47:4265 , piece_id=G3UNZA6XFSSLHWYS5SKZ7P3IM5GQBZACNIEJJZTYTG5UM5IRZ66A, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.621416848 (515h58m30.492499894s) [noise=true , address=145.79.227.8:4330 , piece_id=OEEEMUWMDQLCLRT7P3CPAD3RXQD2WQD256OE3RKSE7UPKLHXJTDQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.792748072 (515h58m30.663831107s) [noise=true , address=45.91.116.10:4083 , piece_id=ONSNWUB3WA4JOEOUTUGQOD54YVE4DFNKA4VH7GPDGBHYFUEGIJ4A, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.819862232 (515h58m30.690945277s) [noise=true , address=45.91.116.11:4120 , piece_id=64RL6MY43NZ6IKR32HXD3VAX434T6I7UDH72QFRQJLXT25QMJHYA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.856879379 (515h58m30.727962424s) [noise=true , address=45.91.116.9:4261 , piece_id=QESJQZSRVGCTZZDKKEDHTPBFHCIKI2WOMIFQTAFOIH3HDP7U7JOA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.857410149 (515h58m30.728493184s) [noise=true , address=45.91.116.12:4226 , piece_id=2PVJKD2URAJ4CEYGQTMPY6DO6XU3V4OLDFYLUSNRK2DLBEAT5YVQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.860047942 (515h58m30.731130977s) [noise=true , address=77.90.31.11:4554 , piece_id=OEBFHSK6V5V5IMDNX3AKQPWVFCGOBDOEFJI37VHRSCVBRJUTW5CQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: <nil> Dec 27 03:27:30.860061588 (515h58m30.731144633s) [seg_pos={0 0}, upload=452732]: Segment reached optimal threshold of 13 pieces. Dec 27 03:27:30.860271858 (515h58m30.731354893s) [noise=true , address=145.79.227.30:4761 , piece_id=FEZCPQFHCZDECIGPDLUB667YGROCXJ4MS67CLO243LQZGILEPHWQ, seg_pos={0 0}, upload=452732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 03:27:30.860282198 (515h58m30.731365234s) [noise=true , address=145.79.227.30:4761 , piece_id=FEZCPQFHCZDECIGPDLUB667YGROCXJ4MS67CLO243LQZGILEPHWQ, seg_pos={0 0}, upload=452732]: Long tail cancellation due to optimal threshold Dec 27 03:27:30.860398540 (515h58m30.731481585s) [noise=true , address=145.79.227.35:4481 , piece_id=DIYCBOGZ36F5QIGGVIYIEDSUZEZDXNK3W6ZBB3SLRYG5S6GOMPIA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 27 03:27:30.860412918 (515h58m30.731495953s) [noise=true , address=145.79.227.35:4481 , piece_id=DIYCBOGZ36F5QIGGVIYIEDSUZEZDXNK3W6ZBB3SLRYG5S6GOMPIA, seg_pos={0 0}, upload=452732]: Long tail cancellation due to optimal threshold Dec 27 03:27:30.860500885 (515h58m30.73158392s) [noise=true , address=45.91.116.13:4155 , piece_id=P6XKCDK4X7I3XDD7PU5B4QJUWQ7VL6WXB77LZVU6DINTZILHBSUA, seg_pos={0 0}, upload=452732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 03:27:30.860506896 (515h58m30.731589941s) [noise=true , address=45.91.116.13:4155 , piece_id=P6XKCDK4X7I3XDD7PU5B4QJUWQ7VL6WXB77LZVU6DINTZILHBSUA, seg_pos={0 0}, upload=452732]: Long tail cancellation due to optimal threshold Dec 27 03:27:30.860522697 (515h58m30.731605732s) [seg_pos={0 0}, upload=452732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:27:30.860586447 (515h58m30.731669492s) [upload=452732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:27:30.860591067 (515h58m30.731674102s) [upload=452732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:27:31.155005780 (515h58m31.026088815s) [upload=452732]: Done uploading segments... Dec 27 03:27:31.155026508 (515h58m31.026109554s) [upload=452732]: Upload finished. err: <nil> Duration: 30.003974678s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX.] duration: 6.062773798s trace: https://jaeger.dev.storj.io/trace/27b6580f3f24d9f5 FAIL (2025-12-27 03:27:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: HHTRH6BRM5RDK6ZO56MHPSCU2ZAMCUQ5F2BZUKSM3GM3SH5NEEKQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: BOSEZHSHQ6VQ3PC3XV6S5WBKRG4XFDGJ3RJARNGUOCOR5KT4YJDA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: TE2FHHOADQWFBQ4HWEOR4NMCTAZBQDMQAFJ6FGNAYOL2WEC7YZYA) context deadline exceeded ======LOGS====== Dec 27 03:26:03.575263221 (515h57m3.446346266s) [upload=452711]: Starting upload Dec 27 03:26:03.575327323 (515h57m3.446410358s) [upload=452711]: Done starting upload Dec 27 03:26:03.575455808 (515h57m3.446538843s) [upload=452711]: Uploading segments... Dec 27 03:26:03.575550878 (515h57m3.446633923s) [upload=452711]: Got next segment. Inline: false Dec 27 03:26:03.575699402 (515h57m3.446782447s) [upload=452711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:26:03.575707818 (515h57m3.446790863s) [upload=452711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:26:03.816559991 (515h57m3.687643026s) [seg_pos={0 0}, upload=452711]: Begin upload segment... Dec 27 03:26:03.816616979 (515h57m3.687700024s) [seg_pos={0 0}, upload=452711]: Done begin upload segment. Dec 27 03:26:03.816665351 (515h57m3.687748396s) [noise=true , address=45.91.116.8:4047 , piece_id=BOSEZHSHQ6VQ3PC3XV6S5WBKRG4XFDGJ3RJARNGUOCOR5KT4YJDA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.816678907 (515h57m3.687761942s) [noise=true , address=145.79.227.34:4906 , piece_id=T6T5OQMWQN5NHVGO5CI2JGJONQZJWRUSQTHVM4HFOBM6PSZZ7S3A, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.817350806 (515h57m3.688433841s) [noise=true , address=145.79.227.8:4330 , piece_id=VJFRNG2EBL6QII26UCOPM2WK2T6JJ5XJ5CDZDUDYKKG4OEOSWZDQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.817688038 (515h57m3.688771083s) [noise=true , address=145.79.227.19:4079 , piece_id=BQEN7T6KLLG34KEWDBCLRKPS3XMDH6R2B45OGU6PKPTXT4E74BKQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.818856163 (515h57m3.689939218s) [noise=true , address=45.91.116.12:4227 , piece_id=HHTRH6BRM5RDK6ZO56MHPSCU2ZAMCUQ5F2BZUKSM3GM3SH5NEEKQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.819796604 (515h57m3.690879649s) [noise=true , address=145.79.227.17:4970 , piece_id=H6SBB6RF3BX7UA4UOOBJBX4G5OTMXO7KWGG62MGTSRVTXGST332Q, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.820816987 (515h57m3.691900022s) [noise=true , address=77.90.31.21:4878 , piece_id=WBJVPEVJ2LB5GEUKJBMJ7GSL52VRNBF5NYLKT3JHFBTKOYTB2WLA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.821305837 (515h57m3.692388882s) [noise=true , address=145.79.227.13:4803 , piece_id=KXV2FPRLVB4UAVBJJ3VSVQ2DXQIT2HKEX7JFU6FRAAO7NK7VLEBQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.823012437 (515h57m3.694095482s) [noise=true , address=45.91.116.7:4011 , piece_id=TE2FHHOADQWFBQ4HWEOR4NMCTAZBQDMQAFJ6FGNAYOL2WEC7YZYA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.824583689 (515h57m3.695666734s) [noise=true , address=145.79.227.39:4014 , piece_id=QQHMG2RQEKFPOUNCAPPIVMQ27WITYXLMTK6I42C2L5AIHU6T3YPA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.826053639 (515h57m3.697136684s) [noise=true , address=145.79.227.42:4123 , piece_id=J5PFYA5X4R2TTKGDOKV7GO66TBOZT5U2D5QNWEH3DO6SPY6O5VXA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.826971015 (515h57m3.69805406s) [noise=true , address=145.79.227.28:4727 , piece_id=5CVWI2UVR5WQ3ZKXYT7DQ5ESLDPFE7K73UV6APTLNNXIUDVXARPA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.828660812 (515h57m3.699743847s) [noise=true , address=145.79.227.41:4086 , piece_id=PDUGXMSZQYGFEYLLA6PHSVU5J6GB3DFDXPWESWG64SJTG47TLJDQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.829520079 (515h57m3.700603124s) [noise=true , address=45.91.116.11:4118 , piece_id=I65X6VXLZF3S5ODPIFH4VH4PTWJSTMXL5OAS6UTIK2AQGTNB6B4Q, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.830864088 (515h57m3.701947133s) [noise=true , address=145.79.227.9:4365 , piece_id=7FUW4ZPKFOUHAQOFLIVV5QU47KHPJRTOTITTH5M7ARVYI4OANXFA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.831699339 (515h57m3.702782384s) [noise=true , address=77.90.31.11:4556 , piece_id=HOAWKXKAOCSSTKY7FRTAY25IXYAG2EIIOKPCKHKZN2INAWPOZ7UQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:05.532010098 (515h57m5.403093143s) [upload=452711]: Next returned nil segment Dec 27 03:26:05.532061946 (515h57m5.403144991s) [upload=452711]: Waiting for error group managing segments... Dec 27 03:26:05.813167800 (515h57m5.684250835s) [noise=true , address=145.79.227.8:4330 , piece_id=VJFRNG2EBL6QII26UCOPM2WK2T6JJ5XJ5CDZDUDYKKG4OEOSWZDQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.831305196 (515h57m5.702388241s) [noise=true , address=145.79.227.9:4365 , piece_id=7FUW4ZPKFOUHAQOFLIVV5QU47KHPJRTOTITTH5M7ARVYI4OANXFA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.846042769 (515h57m5.717125804s) [noise=true , address=145.79.227.28:4727 , piece_id=5CVWI2UVR5WQ3ZKXYT7DQ5ESLDPFE7K73UV6APTLNNXIUDVXARPA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.848213091 (515h57m5.719296136s) [noise=true , address=145.79.227.17:4970 , piece_id=H6SBB6RF3BX7UA4UOOBJBX4G5OTMXO7KWGG62MGTSRVTXGST332Q, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.848509125 (515h57m5.71959216s) [noise=true , address=145.79.227.13:4803 , piece_id=KXV2FPRLVB4UAVBJJ3VSVQ2DXQIT2HKEX7JFU6FRAAO7NK7VLEBQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.850736627 (515h57m5.721819672s) [noise=true , address=145.79.227.19:4079 , piece_id=BQEN7T6KLLG34KEWDBCLRKPS3XMDH6R2B45OGU6PKPTXT4E74BKQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.852169395 (515h57m5.72325243s) [noise=true , address=145.79.227.39:4014 , piece_id=QQHMG2RQEKFPOUNCAPPIVMQ27WITYXLMTK6I42C2L5AIHU6T3YPA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.852775579 (515h57m5.723858624s) [noise=true , address=145.79.227.42:4123 , piece_id=J5PFYA5X4R2TTKGDOKV7GO66TBOZT5U2D5QNWEH3DO6SPY6O5VXA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.855143007 (515h57m5.726226052s) [noise=true , address=145.79.227.41:4086 , piece_id=PDUGXMSZQYGFEYLLA6PHSVU5J6GB3DFDXPWESWG64SJTG47TLJDQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:06.026009295 (515h57m5.89709234s) [noise=true , address=77.90.31.21:4878 , piece_id=WBJVPEVJ2LB5GEUKJBMJ7GSL52VRNBF5NYLKT3JHFBTKOYTB2WLA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:06.054994769 (515h57m5.926077804s) [noise=true , address=45.91.116.8:4047 , piece_id=BOSEZHSHQ6VQ3PC3XV6S5WBKRG4XFDGJ3RJARNGUOCOR5KT4YJDA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:06.082274524 (515h57m5.953357569s) [noise=true , address=45.91.116.12:4227 , piece_id=HHTRH6BRM5RDK6ZO56MHPSCU2ZAMCUQ5F2BZUKSM3GM3SH5NEEKQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:06.094864878 (515h57m5.965947913s) [noise=true , address=45.91.116.7:4011 , piece_id=TE2FHHOADQWFBQ4HWEOR4NMCTAZBQDMQAFJ6FGNAYOL2WEC7YZYA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:06.094891389 (515h57m5.965974434s) [seg_pos={0 0}, upload=452711]: Segment reached optimal threshold of 13 pieces. Dec 27 03:26:06.095171562 (515h57m5.966254608s) [noise=true , address=45.91.116.11:4118 , piece_id=I65X6VXLZF3S5ODPIFH4VH4PTWJSTMXL5OAS6UTIK2AQGTNB6B4Q, seg_pos={0 0}, upload=452711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 27 03:26:06.095180900 (515h57m5.966263945s) [noise=true , address=45.91.116.11:4118 , piece_id=I65X6VXLZF3S5ODPIFH4VH4PTWJSTMXL5OAS6UTIK2AQGTNB6B4Q, seg_pos={0 0}, upload=452711]: Long tail cancellation due to optimal threshold Dec 27 03:26:06.095399637 (515h57m5.966482682s) [noise=true , address=145.79.227.34:4906 , piece_id=T6T5OQMWQN5NHVGO5CI2JGJONQZJWRUSQTHVM4HFOBM6PSZZ7S3A, seg_pos={0 0}, upload=452711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 27 03:26:06.095418863 (515h57m5.966501908s) [noise=true , address=145.79.227.34:4906 , piece_id=T6T5OQMWQN5NHVGO5CI2JGJONQZJWRUSQTHVM4HFOBM6PSZZ7S3A, seg_pos={0 0}, upload=452711]: Long tail cancellation due to optimal threshold Dec 27 03:26:06.095540125 (515h57m5.96662316s) [noise=true , address=77.90.31.11:4556 , piece_id=HOAWKXKAOCSSTKY7FRTAY25IXYAG2EIIOKPCKHKZN2INAWPOZ7UQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:35420->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:35420->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35420->77.90.31.11:4556: use of closed network connection Dec 27 03:26:06.095550504 (515h57m5.966633539s) [noise=true , address=77.90.31.11:4556 , piece_id=HOAWKXKAOCSSTKY7FRTAY25IXYAG2EIIOKPCKHKZN2INAWPOZ7UQ, seg_pos={0 0}, upload=452711]: Long tail cancellation due to optimal threshold Dec 27 03:26:06.095577255 (515h57m5.9666603s) [seg_pos={0 0}, upload=452711]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:26:06.095665363 (515h57m5.966748398s) [upload=452711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:26:06.095671905 (515h57m5.96675495s) [upload=452711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:26:06.387429571 (515h57m6.258512626s) [upload=452711]: Done uploading segments... Dec 27 03:26:06.387455210 (515h57m6.258538265s) [upload=452711]: Upload finished. err: <nil> Duration: 6.062805389s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXX.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XXX] duration: 30.003015977s trace: https://jaeger.dev.storj.io/trace/27b6580f3f24d9f5 FAIL (2025-12-27 03:26:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: HHTRH6BRM5RDK6ZO56MHPSCU2ZAMCUQ5F2BZUKSM3GM3SH5NEEKQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: BOSEZHSHQ6VQ3PC3XV6S5WBKRG4XFDGJ3RJARNGUOCOR5KT4YJDA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: TE2FHHOADQWFBQ4HWEOR4NMCTAZBQDMQAFJ6FGNAYOL2WEC7YZYA) context deadline exceeded ======LOGS====== Dec 27 03:26:03.575263221 (515h57m3.446346266s) [upload=452711]: Starting upload Dec 27 03:26:03.575327323 (515h57m3.446410358s) [upload=452711]: Done starting upload Dec 27 03:26:03.575455808 (515h57m3.446538843s) [upload=452711]: Uploading segments... Dec 27 03:26:03.575550878 (515h57m3.446633923s) [upload=452711]: Got next segment. Inline: false Dec 27 03:26:03.575699402 (515h57m3.446782447s) [upload=452711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:26:03.575707818 (515h57m3.446790863s) [upload=452711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:26:03.816559991 (515h57m3.687643026s) [seg_pos={0 0}, upload=452711]: Begin upload segment... Dec 27 03:26:03.816616979 (515h57m3.687700024s) [seg_pos={0 0}, upload=452711]: Done begin upload segment. Dec 27 03:26:03.816665351 (515h57m3.687748396s) [noise=true , address=45.91.116.8:4047 , piece_id=BOSEZHSHQ6VQ3PC3XV6S5WBKRG4XFDGJ3RJARNGUOCOR5KT4YJDA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.816678907 (515h57m3.687761942s) [noise=true , address=145.79.227.34:4906 , piece_id=T6T5OQMWQN5NHVGO5CI2JGJONQZJWRUSQTHVM4HFOBM6PSZZ7S3A, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.817350806 (515h57m3.688433841s) [noise=true , address=145.79.227.8:4330 , piece_id=VJFRNG2EBL6QII26UCOPM2WK2T6JJ5XJ5CDZDUDYKKG4OEOSWZDQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.817688038 (515h57m3.688771083s) [noise=true , address=145.79.227.19:4079 , piece_id=BQEN7T6KLLG34KEWDBCLRKPS3XMDH6R2B45OGU6PKPTXT4E74BKQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.818856163 (515h57m3.689939218s) [noise=true , address=45.91.116.12:4227 , piece_id=HHTRH6BRM5RDK6ZO56MHPSCU2ZAMCUQ5F2BZUKSM3GM3SH5NEEKQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.819796604 (515h57m3.690879649s) [noise=true , address=145.79.227.17:4970 , piece_id=H6SBB6RF3BX7UA4UOOBJBX4G5OTMXO7KWGG62MGTSRVTXGST332Q, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.820816987 (515h57m3.691900022s) [noise=true , address=77.90.31.21:4878 , piece_id=WBJVPEVJ2LB5GEUKJBMJ7GSL52VRNBF5NYLKT3JHFBTKOYTB2WLA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.821305837 (515h57m3.692388882s) [noise=true , address=145.79.227.13:4803 , piece_id=KXV2FPRLVB4UAVBJJ3VSVQ2DXQIT2HKEX7JFU6FRAAO7NK7VLEBQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.823012437 (515h57m3.694095482s) [noise=true , address=45.91.116.7:4011 , piece_id=TE2FHHOADQWFBQ4HWEOR4NMCTAZBQDMQAFJ6FGNAYOL2WEC7YZYA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.824583689 (515h57m3.695666734s) [noise=true , address=145.79.227.39:4014 , piece_id=QQHMG2RQEKFPOUNCAPPIVMQ27WITYXLMTK6I42C2L5AIHU6T3YPA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.826053639 (515h57m3.697136684s) [noise=true , address=145.79.227.42:4123 , piece_id=J5PFYA5X4R2TTKGDOKV7GO66TBOZT5U2D5QNWEH3DO6SPY6O5VXA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.826971015 (515h57m3.69805406s) [noise=true , address=145.79.227.28:4727 , piece_id=5CVWI2UVR5WQ3ZKXYT7DQ5ESLDPFE7K73UV6APTLNNXIUDVXARPA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.828660812 (515h57m3.699743847s) [noise=true , address=145.79.227.41:4086 , piece_id=PDUGXMSZQYGFEYLLA6PHSVU5J6GB3DFDXPWESWG64SJTG47TLJDQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.829520079 (515h57m3.700603124s) [noise=true , address=45.91.116.11:4118 , piece_id=I65X6VXLZF3S5ODPIFH4VH4PTWJSTMXL5OAS6UTIK2AQGTNB6B4Q, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.830864088 (515h57m3.701947133s) [noise=true , address=145.79.227.9:4365 , piece_id=7FUW4ZPKFOUHAQOFLIVV5QU47KHPJRTOTITTH5M7ARVYI4OANXFA, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:03.831699339 (515h57m3.702782384s) [noise=true , address=77.90.31.11:4556 , piece_id=HOAWKXKAOCSSTKY7FRTAY25IXYAG2EIIOKPCKHKZN2INAWPOZ7UQ, seg_pos={0 0}, upload=452711]: Uploading piece... Dec 27 03:26:05.532010098 (515h57m5.403093143s) [upload=452711]: Next returned nil segment Dec 27 03:26:05.532061946 (515h57m5.403144991s) [upload=452711]: Waiting for error group managing segments... Dec 27 03:26:05.813167800 (515h57m5.684250835s) [noise=true , address=145.79.227.8:4330 , piece_id=VJFRNG2EBL6QII26UCOPM2WK2T6JJ5XJ5CDZDUDYKKG4OEOSWZDQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.831305196 (515h57m5.702388241s) [noise=true , address=145.79.227.9:4365 , piece_id=7FUW4ZPKFOUHAQOFLIVV5QU47KHPJRTOTITTH5M7ARVYI4OANXFA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.846042769 (515h57m5.717125804s) [noise=true , address=145.79.227.28:4727 , piece_id=5CVWI2UVR5WQ3ZKXYT7DQ5ESLDPFE7K73UV6APTLNNXIUDVXARPA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.848213091 (515h57m5.719296136s) [noise=true , address=145.79.227.17:4970 , piece_id=H6SBB6RF3BX7UA4UOOBJBX4G5OTMXO7KWGG62MGTSRVTXGST332Q, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.848509125 (515h57m5.71959216s) [noise=true , address=145.79.227.13:4803 , piece_id=KXV2FPRLVB4UAVBJJ3VSVQ2DXQIT2HKEX7JFU6FRAAO7NK7VLEBQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.850736627 (515h57m5.721819672s) [noise=true , address=145.79.227.19:4079 , piece_id=BQEN7T6KLLG34KEWDBCLRKPS3XMDH6R2B45OGU6PKPTXT4E74BKQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.852169395 (515h57m5.72325243s) [noise=true , address=145.79.227.39:4014 , piece_id=QQHMG2RQEKFPOUNCAPPIVMQ27WITYXLMTK6I42C2L5AIHU6T3YPA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.852775579 (515h57m5.723858624s) [noise=true , address=145.79.227.42:4123 , piece_id=J5PFYA5X4R2TTKGDOKV7GO66TBOZT5U2D5QNWEH3DO6SPY6O5VXA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:05.855143007 (515h57m5.726226052s) [noise=true , address=145.79.227.41:4086 , piece_id=PDUGXMSZQYGFEYLLA6PHSVU5J6GB3DFDXPWESWG64SJTG47TLJDQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:06.026009295 (515h57m5.89709234s) [noise=true , address=77.90.31.21:4878 , piece_id=WBJVPEVJ2LB5GEUKJBMJ7GSL52VRNBF5NYLKT3JHFBTKOYTB2WLA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:06.054994769 (515h57m5.926077804s) [noise=true , address=45.91.116.8:4047 , piece_id=BOSEZHSHQ6VQ3PC3XV6S5WBKRG4XFDGJ3RJARNGUOCOR5KT4YJDA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:06.082274524 (515h57m5.953357569s) [noise=true , address=45.91.116.12:4227 , piece_id=HHTRH6BRM5RDK6ZO56MHPSCU2ZAMCUQ5F2BZUKSM3GM3SH5NEEKQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:06.094864878 (515h57m5.965947913s) [noise=true , address=45.91.116.7:4011 , piece_id=TE2FHHOADQWFBQ4HWEOR4NMCTAZBQDMQAFJ6FGNAYOL2WEC7YZYA, seg_pos={0 0}, upload=452711]: Done uploading piece. err: <nil> Dec 27 03:26:06.094891389 (515h57m5.965974434s) [seg_pos={0 0}, upload=452711]: Segment reached optimal threshold of 13 pieces. Dec 27 03:26:06.095171562 (515h57m5.966254608s) [noise=true , address=45.91.116.11:4118 , piece_id=I65X6VXLZF3S5ODPIFH4VH4PTWJSTMXL5OAS6UTIK2AQGTNB6B4Q, seg_pos={0 0}, upload=452711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 27 03:26:06.095180900 (515h57m5.966263945s) [noise=true , address=45.91.116.11:4118 , piece_id=I65X6VXLZF3S5ODPIFH4VH4PTWJSTMXL5OAS6UTIK2AQGTNB6B4Q, seg_pos={0 0}, upload=452711]: Long tail cancellation due to optimal threshold Dec 27 03:26:06.095399637 (515h57m5.966482682s) [noise=true , address=145.79.227.34:4906 , piece_id=T6T5OQMWQN5NHVGO5CI2JGJONQZJWRUSQTHVM4HFOBM6PSZZ7S3A, seg_pos={0 0}, upload=452711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 27 03:26:06.095418863 (515h57m5.966501908s) [noise=true , address=145.79.227.34:4906 , piece_id=T6T5OQMWQN5NHVGO5CI2JGJONQZJWRUSQTHVM4HFOBM6PSZZ7S3A, seg_pos={0 0}, upload=452711]: Long tail cancellation due to optimal threshold Dec 27 03:26:06.095540125 (515h57m5.96662316s) [noise=true , address=77.90.31.11:4556 , piece_id=HOAWKXKAOCSSTKY7FRTAY25IXYAG2EIIOKPCKHKZN2INAWPOZ7UQ, seg_pos={0 0}, upload=452711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:35420->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:35420->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35420->77.90.31.11:4556: use of closed network connection Dec 27 03:26:06.095550504 (515h57m5.966633539s) [noise=true , address=77.90.31.11:4556 , piece_id=HOAWKXKAOCSSTKY7FRTAY25IXYAG2EIIOKPCKHKZN2INAWPOZ7UQ, seg_pos={0 0}, upload=452711]: Long tail cancellation due to optimal threshold Dec 27 03:26:06.095577255 (515h57m5.9666603s) [seg_pos={0 0}, upload=452711]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:26:06.095665363 (515h57m5.966748398s) [upload=452711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:26:06.095671905 (515h57m5.96675495s) [upload=452711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:26:06.387429571 (515h57m6.258512626s) [upload=452711]: Done uploading segments... Dec 27 03:26:06.387455210 (515h57m6.258538265s) [upload=452711]: Upload finished. err: <nil> Duration: 30.003158258s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XXX.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.XX] duration: 30.003301934s trace: https://jaeger.dev.storj.io/trace/e3349be50e79a15 FAIL (2025-12-27 03:26:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: Z4ON4VW4CYRKQVKZMOKSSQVJGSXAVJCZNUEPGUC77WFBE4SYLWWQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: O5VP623UBH5ZVXM5NEEYGZXBC65K2ZWWDY3Y3MU35TOINOTVGD3A) context deadline exceeded ======LOGS====== 3:25:33.358036519 (515h56m33.229119574s) [upload=452704]: Uploading segments... Dec 27 03:25:33.358293118 (515h56m33.229376163s) [upload=452704]: Got next segment. Inline: false Dec 27 03:25:33.358342251 (515h56m33.229425296s) [upload=452704]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:25:33.358353332 (515h56m33.229436367s) [upload=452704]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:25:33.606939954 (515h56m33.478022989s) [seg_pos={0 0}, upload=452704]: Begin upload segment... Dec 27 03:25:33.606993816 (515h56m33.478076851s) [seg_pos={0 0}, upload=452704]: Done begin upload segment. Dec 27 03:25:33.607024594 (515h56m33.478107629s) [noise=true , address=145.79.227.11:4732 , piece_id=D4PC5S6TWBVQW4RIBVOVTYNHRE4BQDCOTB3JD6XT6GHNPHB4WIIQ, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.607082986 (515h56m33.478166091s) [noise=true , address=145.79.227.24:4582 , piece_id=SYBISFKB2B4RD5U4IO7GPIAT6U7GBOXPBZEU7XZDBEMYUOWTIYHQ, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.607870374 (515h56m33.47895341s) [noise=true , address=45.91.116.14:4190 , piece_id=IMPYMWKGDZ4FVBX7XWNFZLLZZO3BFMFVWHHIGSAEWAPFIE2JNRHQ, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.608463233 (515h56m33.479546278s) [noise=true , address=145.79.227.41:4087 , piece_id=6JIHYLRINSXS3HTBH22ONW62I7S5XUJ5INOVS5GOFFJS5GU6EPFQ, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.609158738 (515h56m33.480241783s) [noise=true , address=77.90.31.19:4841 , piece_id=AIWBAOOO7UEOOPKB524MPGO4VIBHENCKRO2RLQ33JCDRFCPS47QQ, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.609607603 (515h56m33.480690648s) [noise=true , address=45.91.116.8:4048 , piece_id=YCW33SDTPT7I2IPUAHYH4SMQA7TESZN3BGNU5IU5A7Y6JDMNPDJQ, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.609635175 (515h56m33.48071822s) [noise=true , address=45.91.116.5:4297 , piece_id=W2MUIG4QPOFGGH3OUYCKNMZG5A72GTDDUP23AFF5K6WTXR4WDELA, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.610077668 (515h56m33.481160703s) [noise=true , address=145.79.227.2:4115 , piece_id=WWYJ5RUWC7D222ETM5IAOT4ZD3CSWVQQYXEZJG4WIK3PZGWSIT7Q, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.610328265 (515h56m33.48141131s) [noise=true , address=145.79.227.9:4366 , piece_id=AADFW4M434XYGYN7JZTTMRAEDZ56B5QNKAQ535ZJG7SELMSFCGNA, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.610573562 (515h56m33.481656607s) [noise=true , address=77.90.31.11:4553 , piece_id=Z4ON4VW4CYRKQVKZMOKSSQVJGSXAVJCZNUEPGUC77WFBE4SYLWWQ, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.611013871 (515h56m33.482096906s) [noise=true , address=45.91.116.4:4940 , piece_id=OHJLOKV36R7NADVHKQ6PGEO36GPFNUNWQ5EGWP74W5YRZYDQ6K7A, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.611093442 (515h56m33.482176487s) [noise=true , address=145.79.227.19:4077 , piece_id=2I26UQAVWJ3OLSXTZH2W663KGRE2BAJGSBGL47VWDREPRZE6LXGQ, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.611371231 (515h56m33.482454276s) [noise=true , address=45.91.116.6:4976 , piece_id=O5VP623UBH5ZVXM5NEEYGZXBC65K2ZWWDY3Y3MU35TOINOTVGD3A, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.611739292 (515h56m33.482822337s) [noise=true , address=145.79.227.45:4231 , piece_id=2APZL267BRTFD66E3RMYISXACBDY4VJRQZ43BCKRZQW2RDF4TD4A, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.612266466 (515h56m33.483349501s) [noise=true , address=145.79.227.47:4265 , piece_id=37BZFSGA6J6GNE7RPVRPJT5NCDH7ZE6CQIWP472KP2A3ZOY77RYQ, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:33.614340535 (515h56m33.48542358s) [noise=true , address=145.79.227.51:4409 , piece_id=TTIT3E6XJOZP2SM7CZMGATJUIUQ5B5T7ZZM2EDDZGXNOVSBLFISQ, seg_pos={0 0}, upload=452704]: Uploading piece... Dec 27 03:25:35.316105834 (515h56m35.187188869s) [upload=452704]: Next returned nil segment Dec 27 03:25:35.316119400 (515h56m35.187202435s) [upload=452704]: Waiting for error group managing segments... Dec 27 03:25:35.617957640 (515h56m35.489040675s) [noise=true , address=145.79.227.24:4582 , piece_id=SYBISFKB2B4RD5U4IO7GPIAT6U7GBOXPBZEU7XZDBEMYUOWTIYHQ, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.620628657 (515h56m35.491711692s) [noise=true , address=145.79.227.45:4231 , piece_id=2APZL267BRTFD66E3RMYISXACBDY4VJRQZ43BCKRZQW2RDF4TD4A, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.624750074 (515h56m35.495833109s) [noise=true , address=145.79.227.11:4732 , piece_id=D4PC5S6TWBVQW4RIBVOVTYNHRE4BQDCOTB3JD6XT6GHNPHB4WIIQ, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.633539141 (515h56m35.504622186s) [noise=true , address=145.79.227.9:4366 , piece_id=AADFW4M434XYGYN7JZTTMRAEDZ56B5QNKAQ535ZJG7SELMSFCGNA, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.638175008 (515h56m35.509258043s) [noise=true , address=145.79.227.51:4409 , piece_id=TTIT3E6XJOZP2SM7CZMGATJUIUQ5B5T7ZZM2EDDZGXNOVSBLFISQ, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.649548475 (515h56m35.52063152s) [noise=true , address=145.79.227.41:4087 , piece_id=6JIHYLRINSXS3HTBH22ONW62I7S5XUJ5INOVS5GOFFJS5GU6EPFQ, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.650622230 (515h56m35.521705265s) [noise=true , address=145.79.227.47:4265 , piece_id=37BZFSGA6J6GNE7RPVRPJT5NCDH7ZE6CQIWP472KP2A3ZOY77RYQ, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.656319227 (515h56m35.527402272s) [noise=true , address=145.79.227.19:4077 , piece_id=2I26UQAVWJ3OLSXTZH2W663KGRE2BAJGSBGL47VWDREPRZE6LXGQ, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.849196800 (515h56m35.720279835s) [noise=true , address=45.91.116.8:4048 , piece_id=YCW33SDTPT7I2IPUAHYH4SMQA7TESZN3BGNU5IU5A7Y6JDMNPDJQ, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.871443081 (515h56m35.742526116s) [noise=true , address=45.91.116.5:4297 , piece_id=W2MUIG4QPOFGGH3OUYCKNMZG5A72GTDDUP23AFF5K6WTXR4WDELA, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.892759070 (515h56m35.763842115s) [noise=true , address=77.90.31.11:4553 , piece_id=Z4ON4VW4CYRKQVKZMOKSSQVJGSXAVJCZNUEPGUC77WFBE4SYLWWQ, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:35.911393342 (515h56m35.782476377s) [noise=true , address=45.91.116.6:4976 , piece_id=O5VP623UBH5ZVXM5NEEYGZXBC65K2ZWWDY3Y3MU35TOINOTVGD3A, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:36.010997385 (515h56m35.88208042s) [noise=true , address=45.91.116.4:4940 , piece_id=OHJLOKV36R7NADVHKQ6PGEO36GPFNUNWQ5EGWP74W5YRZYDQ6K7A, seg_pos={0 0}, upload=452704]: Done uploading piece. err: <nil> Dec 27 03:25:36.011018235 (515h56m35.88210128s) [seg_pos={0 0}, upload=452704]: Segment reached optimal threshold of 13 pieces. Dec 27 03:25:36.011218356 (515h56m35.882301391s) [noise=true , address=45.91.116.14:4190 , piece_id=IMPYMWKGDZ4FVBX7XWNFZLLZZO3BFMFVWHHIGSAEWAPFIE2JNRHQ, seg_pos={0 0}, upload=452704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:38104->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:38104->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38104->45.91.116.14:4190: use of closed network connection Dec 27 03:25:36.011228075 (515h56m35.88231111s) [noise=true , address=45.91.116.14:4190 , piece_id=IMPYMWKGDZ4FVBX7XWNFZLLZZO3BFMFVWHHIGSAEWAPFIE2JNRHQ, seg_pos={0 0}, upload=452704]: Long tail cancellation due to optimal threshold Dec 27 03:25:36.011304450 (515h56m35.882387495s) [noise=true , address=77.90.31.19:4841 , piece_id=AIWBAOOO7UEOOPKB524MPGO4VIBHENCKRO2RLQ33JCDRFCPS47QQ, seg_pos={0 0}, upload=452704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 27 03:25:36.011321572 (515h56m35.882404607s) [noise=true , address=77.90.31.19:4841 , piece_id=AIWBAOOO7UEOOPKB524MPGO4VIBHENCKRO2RLQ33JCDRFCPS47QQ, seg_pos={0 0}, upload=452704]: Long tail cancellation due to optimal threshold Dec 27 03:25:36.011532704 (515h56m35.882615749s) [noise=true , address=145.79.227.2:4115 , piece_id=WWYJ5RUWC7D222ETM5IAOT4ZD3CSWVQQYXEZJG4WIK3PZGWSIT7Q, seg_pos={0 0}, upload=452704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:54018->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:54018->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: context canceled Dec 27 03:25:36.011541051 (515h56m35.882624086s) [noise=true , address=145.79.227.2:4115 , piece_id=WWYJ5RUWC7D222ETM5IAOT4ZD3CSWVQQYXEZJG4WIK3PZGWSIT7Q, seg_pos={0 0}, upload=452704]: Long tail cancellation due to optimal threshold Dec 27 03:25:36.011557401 (515h56m35.882640436s) [seg_pos={0 0}, upload=452704]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:25:36.011623067 (515h56m35.882706112s) [upload=452704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:25:36.011627555 (515h56m35.88271061s) [upload=452704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:25:36.309156130 (515h56m36.180239175s) [upload=452704]: Done uploading segments... Dec 27 03:25:36.309175617 (515h56m36.180258662s) [upload=452704]: Upload finished. err: <nil> Duration: 30.003320989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [X.XXX.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.X] duration: 30.004106247s trace: https://jaeger.dev.storj.io/trace/3e7fc2377feaef73 FAIL (2025-12-27 03:25:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: VGDO3ELBZTN45BT5EZMHWIWAWV4OM66AGXYFUTLCNNAK4SSBJ7TQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZCWDZ7DRYUM5PDVL52W3AJ4KPBZP6H3CD3UNFNWK7HBDNVSQUXWA) context deadline exceeded ======LOGS====== 515h55m47.291432863s) [upload=452695]: Done starting upload Dec 27 03:24:47.420522116 (515h55m47.291605151s) [upload=452695]: Uploading segments... Dec 27 03:24:47.420569566 (515h55m47.291652601s) [upload=452695]: Got next segment. Inline: false Dec 27 03:24:47.420591338 (515h55m47.291674373s) [upload=452695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:24:47.420595495 (515h55m47.29167853s) [upload=452695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:24:47.665965501 (515h55m47.537048557s) [seg_pos={0 0}, upload=452695]: Begin upload segment... Dec 27 03:24:47.666011660 (515h55m47.537094695s) [seg_pos={0 0}, upload=452695]: Done begin upload segment. Dec 27 03:24:47.666044823 (515h55m47.537127858s) [noise=true , address=145.79.227.37:4567 , piece_id=YDE4YLGSEQANVJVY7AJE4ACSZMDEOGMAELGD2URIRRZKCKDCGVUQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.666099186 (515h55m47.537182221s) [noise=true , address=45.91.116.9:4264 , piece_id=VGDO3ELBZTN45BT5EZMHWIWAWV4OM66AGXYFUTLCNNAK4SSBJ7TQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.666853483 (515h55m47.537936518s) [noise=true , address=145.79.227.12:4765 , piece_id=DUXXONRW5N5QZCZAGHJM4YNTN6YLTLAF43U65V4JY5NB2G7IHA2A, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.667128677 (515h55m47.538211722s) [noise=true , address=145.79.227.51:4410 , piece_id=7P4PSGMRXUFQCUKZA32RESU2BKMR3WGBEPOMEW3AQJC425O2BTOQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.667388621 (515h55m47.538471657s) [noise=true , address=145.79.227.42:4122 , piece_id=HAAKM3AFA6Z67LS5WR7OT426JA5N3WCA4K4EVLPHJX2CLQ4DFYQQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.667653406 (515h55m47.538736451s) [noise=true , address=145.79.227.49:4338 , piece_id=L4KLDUKQBRDXXLXO254HEA5H4EI4J23J7TRLCROUYN2ZABT5CHIQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.667965240 (515h55m47.539048275s) [noise=true , address=45.91.116.5:4298 , piece_id=ZCWDZ7DRYUM5PDVL52W3AJ4KPBZP6H3CD3UNFNWK7HBDNVSQUXWA, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.668231466 (515h55m47.539314501s) [noise=true , address=145.79.227.5:4223 , piece_id=4KQ3GZRAHABGMYUG45LUTFRAHCHYXC2MJVLJ62SMBTPI6T65LXPQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.668519255 (515h55m47.5396023s) [noise=true , address=45.91.116.6:4976 , piece_id=K7MTBFFG4JFZDRAXLNAFSVOW3KAUX7HZT6C7E7LKZJNRP6ANRXNQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.669854197 (515h55m47.540937232s) [noise=true , address=145.79.227.52:4447 , piece_id=W3H3SWOIILIV5H7Z4JS7OX6OJSKAN3DGUO4VR6I6B5L5CAQODQRQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.670926008 (515h55m47.542009043s) [noise=true , address=145.79.227.45:4231 , piece_id=HK6UFNRMVHCU3ZHAN6Y4IYOLFEMCWYHW5ZKSFADK2XGJC3DLS7SQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.671911734 (515h55m47.542994769s) [noise=true , address=145.79.227.6:4259 , piece_id=K6FGHHJGKXLK32WZPCLARRS5RVCWVZUU6RRXDI4JIYOVBE5UB7RQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.672884917 (515h55m47.543967953s) [noise=true , address=45.91.116.4:4940 , piece_id=B3I4EBGGSIT3VJRIGY2VQGH7YA6A2VUBU4VNDMVN4NN7VORQSQXQ, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.673944195 (515h55m47.54502724s) [noise=true , address=45.91.116.7:4010 , piece_id=YJ5K7NTV2IZ4FKSKQCZSRZGPFXMKEZDFEUTCR6C45BPQ2VJE2Y2Q, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.675315907 (515h55m47.546398942s) [noise=true , address=145.79.227.24:4582 , piece_id=QZMDCNYHBONVSPTV5SESR7W3NSVKON4TCU5LW2KREOB5MCQGTB2A, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:47.676830119 (515h55m47.547913155s) [noise=true , address=45.91.116.12:4228 , piece_id=6X4V5IFUUZWXDVXFL4BIOQF57VSAXPE4IQ6QABKPGDWEAC57DGKA, seg_pos={0 0}, upload=452695]: Uploading piece... Dec 27 03:24:49.371042417 (515h55m49.242125452s) [upload=452695]: Next returned nil segment Dec 27 03:24:49.371060150 (515h55m49.242143195s) [upload=452695]: Waiting for error group managing segments... Dec 27 03:24:49.661088863 (515h55m49.532171898s) [noise=true , address=145.79.227.45:4231 , piece_id=HK6UFNRMVHCU3ZHAN6Y4IYOLFEMCWYHW5ZKSFADK2XGJC3DLS7SQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.661947930 (515h55m49.533030975s) [noise=true , address=145.79.227.42:4122 , piece_id=HAAKM3AFA6Z67LS5WR7OT426JA5N3WCA4K4EVLPHJX2CLQ4DFYQQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.666049820 (515h55m49.537132865s) [noise=true , address=145.79.227.51:4410 , piece_id=7P4PSGMRXUFQCUKZA32RESU2BKMR3WGBEPOMEW3AQJC425O2BTOQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.671907594 (515h55m49.542990629s) [noise=true , address=145.79.227.5:4223 , piece_id=4KQ3GZRAHABGMYUG45LUTFRAHCHYXC2MJVLJ62SMBTPI6T65LXPQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.672120510 (515h55m49.543203545s) [noise=true , address=145.79.227.6:4259 , piece_id=K6FGHHJGKXLK32WZPCLARRS5RVCWVZUU6RRXDI4JIYOVBE5UB7RQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.676043119 (515h55m49.547126154s) [noise=true , address=145.79.227.37:4567 , piece_id=YDE4YLGSEQANVJVY7AJE4ACSZMDEOGMAELGD2URIRRZKCKDCGVUQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.679397797 (515h55m49.550480842s) [noise=true , address=145.79.227.52:4447 , piece_id=W3H3SWOIILIV5H7Z4JS7OX6OJSKAN3DGUO4VR6I6B5L5CAQODQRQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.686774944 (515h55m49.557857979s) [noise=true , address=145.79.227.24:4582 , piece_id=QZMDCNYHBONVSPTV5SESR7W3NSVKON4TCU5LW2KREOB5MCQGTB2A, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.687398301 (515h55m49.558481346s) [noise=true , address=145.79.227.49:4338 , piece_id=L4KLDUKQBRDXXLXO254HEA5H4EI4J23J7TRLCROUYN2ZABT5CHIQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.705467428 (515h55m49.576550473s) [noise=true , address=145.79.227.12:4765 , piece_id=DUXXONRW5N5QZCZAGHJM4YNTN6YLTLAF43U65V4JY5NB2G7IHA2A, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.901326778 (515h55m49.772409823s) [noise=true , address=45.91.116.5:4298 , piece_id=ZCWDZ7DRYUM5PDVL52W3AJ4KPBZP6H3CD3UNFNWK7HBDNVSQUXWA, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.918379529 (515h55m49.789462564s) [noise=true , address=45.91.116.4:4940 , piece_id=B3I4EBGGSIT3VJRIGY2VQGH7YA6A2VUBU4VNDMVN4NN7VORQSQXQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.982064786 (515h55m49.853147821s) [noise=true , address=45.91.116.9:4264 , piece_id=VGDO3ELBZTN45BT5EZMHWIWAWV4OM66AGXYFUTLCNNAK4SSBJ7TQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: <nil> Dec 27 03:24:49.982082651 (515h55m49.853165686s) [seg_pos={0 0}, upload=452695]: Segment reached optimal threshold of 13 pieces. Dec 27 03:24:49.982244518 (515h55m49.853327563s) [noise=true , address=45.91.116.6:4976 , piece_id=K7MTBFFG4JFZDRAXLNAFSVOW3KAUX7HZT6C7E7LKZJNRP6ANRXNQ, seg_pos={0 0}, upload=452695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 03:24:49.982251672 (515h55m49.853334707s) [noise=true , address=45.91.116.6:4976 , piece_id=K7MTBFFG4JFZDRAXLNAFSVOW3KAUX7HZT6C7E7LKZJNRP6ANRXNQ, seg_pos={0 0}, upload=452695]: Long tail cancellation due to optimal threshold Dec 27 03:24:49.982400756 (515h55m49.853483791s) [noise=true , address=45.91.116.12:4228 , piece_id=6X4V5IFUUZWXDVXFL4BIOQF57VSAXPE4IQ6QABKPGDWEAC57DGKA, seg_pos={0 0}, upload=452695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:41776->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:41776->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41776->45.91.116.12:4228: use of closed network connection Dec 27 03:24:49.982412458 (515h55m49.853495493s) [noise=true , address=45.91.116.12:4228 , piece_id=6X4V5IFUUZWXDVXFL4BIOQF57VSAXPE4IQ6QABKPGDWEAC57DGKA, seg_pos={0 0}, upload=452695]: Long tail cancellation due to optimal threshold Dec 27 03:24:49.982842657 (515h55m49.853925692s) [noise=true , address=45.91.116.7:4010 , piece_id=YJ5K7NTV2IZ4FKSKQCZSRZGPFXMKEZDFEUTCR6C45BPQ2VJE2Y2Q, seg_pos={0 0}, upload=452695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:55802->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 27 03:24:49.982857326 (515h55m49.853940361s) [noise=true , address=45.91.116.7:4010 , piece_id=YJ5K7NTV2IZ4FKSKQCZSRZGPFXMKEZDFEUTCR6C45BPQ2VJE2Y2Q, seg_pos={0 0}, upload=452695]: Long tail cancellation due to optimal threshold Dec 27 03:24:49.982882985 (515h55m49.85396603s) [seg_pos={0 0}, upload=452695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:24:49.983009976 (515h55m49.854093021s) [upload=452695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:24:49.983014735 (515h55m49.85409777s) [upload=452695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:24:50.275651384 (515h55m50.146734429s) [upload=452695]: Done uploading segments... Dec 27 03:24:50.275669449 (515h55m50.146752494s) [upload=452695]: Upload finished. err: <nil> Duration: 30.004127139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 48.258013751s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:24:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 48.258038988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX.XXX.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X.] duration: 6.123610569s trace: https://jaeger.dev.storj.io/trace/525d751e7b0968da FAIL (2025-12-27 03:23:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GFLPWFM3LAG3LZ6VDPOIYHS7EMAET47YIAQESCV4CD66S7WZQEKA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: B654ECCQPPQP37J534B2NT2BZIWO74QRFP3W22V3JMQFKDEFQYIA) context deadline exceeded ======LOGS====== segments... Dec 27 03:22:47.491017930 (515h53m47.362100975s) [upload=452664]: Got next segment. Inline: false Dec 27 03:22:47.491065400 (515h53m47.362148465s) [upload=452664]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:22:47.491071713 (515h53m47.362154798s) [upload=452664]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:22:47.737055567 (515h53m47.608138602s) [seg_pos={0 0}, upload=452664]: Begin upload segment... Dec 27 03:22:47.737106595 (515h53m47.60818963s) [seg_pos={0 0}, upload=452664]: Done begin upload segment. Dec 27 03:22:47.737146771 (515h53m47.608229816s) [noise=true , address=45.91.116.7:4012 , piece_id=RP6YNLP3ZEWJCFLCPVHHXAVRNF2Q52X4SAZGT7JWMAPFDL4VXEGQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.737221824 (515h53m47.608304859s) [noise=true , address=145.79.227.13:4804 , piece_id=DRBWXBP2FACIDODKAKRTXH7HVXCVXNLPOLZOMAZAUKSK4YIIQ3IA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.737763656 (515h53m47.608846691s) [noise=true , address=145.79.227.38:4979 , piece_id=YWLB24X64WWO7ZSTXCKXH2IPMCXG3ZRDXSONNI4CCG47X3ESCATQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.738206840 (515h53m47.609289875s) [noise=true , address=145.79.227.45:4229 , piece_id=2BQBVCWRJGHNA3GMTQGJDIFLQ33JXDVEQGOOOIBD4EMGOEFI27AQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.738584458 (515h53m47.609667494s) [noise=true , address=145.79.227.33:4870 , piece_id=T43HLNGMWRYV3DXIIN3MAW2P6JNXKRHG4UNXK3EVYO2332VUROXA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.738873550 (515h53m47.609956585s) [noise=true , address=145.79.227.48:4303 , piece_id=GPDMVO2REPZTQHCEKFRJ2GFPJIML6OJREW6I5UBM53YLHPMJTKLQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.739143373 (515h53m47.610226408s) [noise=true , address=45.91.116.5:4300 , piece_id=GFLPWFM3LAG3LZ6VDPOIYHS7EMAET47YIAQESCV4CD66S7WZQEKA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.739504661 (515h53m47.610587706s) [noise=true , address=77.90.31.19:4844 , piece_id=TGJXM437WDPCCPFTCWP342LRB4A4R4HVW7PFZRVLGAB2RC2GX6RQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.740019492 (515h53m47.611102537s) [noise=true , address=77.90.31.14:4663 , piece_id=B654ECCQPPQP37J534B2NT2BZIWO74QRFP3W22V3JMQFKDEFQYIA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.740832980 (515h53m47.611916016s) [noise=true , address=145.79.227.5:4222 , piece_id=DIEYHW5PTSPHVBS7FZRUJ5ATLFX45S7ZABVPYOUDTDA7NBKZTGSQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.742032795 (515h53m47.61311583s) [noise=true , address=145.79.227.15:4874 , piece_id=I2XQWSAKEXC5HK273DSQVUMY5MBRKH33YDKMZ6ZWPQHZHWZ5KANQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.743113472 (515h53m47.614196517s) [noise=true , address=145.79.227.9:4365 , piece_id=5DLCWGLV72EETRYAXBY2Z7A3WBYZHBGFHENUPXAQWBSS776HDNPA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.744484213 (515h53m47.615567248s) [noise=true , address=145.79.227.37:4566 , piece_id=PVEWMPWYO5HYKA7U3PVI5UGNYYM7N6SS433GXPZ2E6P3PRHQC4RA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.746082677 (515h53m47.617165712s) [noise=true , address=145.79.227.51:4409 , piece_id=X7OCV2VP23NYZ7DJIQFLN4AHHFHV5QYVPSUSXKJVPGGCQHGWAH5A, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.749012185 (515h53m47.62009523s) [noise=true , address=77.90.31.15:4700 , piece_id=NPPPGRW6GX2RW4QW7HJGQBKWIPJF62M3ANQ2453VD47N6EIOZ4BA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.752041073 (515h53m47.623124118s) [noise=true , address=145.79.227.8:4330 , piece_id=GIK2VOJHRCTLGRXAMPCCQTB4BUUR465RVFQXQ5BB4D24FWQHYNRA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:49.443921459 (515h53m49.315004494s) [upload=452664]: Next returned nil segment Dec 27 03:22:49.443935365 (515h53m49.3150184s) [upload=452664]: Waiting for error group managing segments... Dec 27 03:22:49.737767541 (515h53m49.608850586s) [noise=true , address=145.79.227.48:4303 , piece_id=GPDMVO2REPZTQHCEKFRJ2GFPJIML6OJREW6I5UBM53YLHPMJTKLQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.738562064 (515h53m49.609645109s) [noise=true , address=145.79.227.8:4330 , piece_id=GIK2VOJHRCTLGRXAMPCCQTB4BUUR465RVFQXQ5BB4D24FWQHYNRA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.754288721 (515h53m49.625371756s) [noise=true , address=145.79.227.15:4874 , piece_id=I2XQWSAKEXC5HK273DSQVUMY5MBRKH33YDKMZ6ZWPQHZHWZ5KANQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.755430825 (515h53m49.62651387s) [noise=true , address=145.79.227.45:4229 , piece_id=2BQBVCWRJGHNA3GMTQGJDIFLQ33JXDVEQGOOOIBD4EMGOEFI27AQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.762945314 (515h53m49.634028349s) [noise=true , address=145.79.227.38:4979 , piece_id=YWLB24X64WWO7ZSTXCKXH2IPMCXG3ZRDXSONNI4CCG47X3ESCATQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.766075334 (515h53m49.637158379s) [noise=true , address=145.79.227.9:4365 , piece_id=5DLCWGLV72EETRYAXBY2Z7A3WBYZHBGFHENUPXAQWBSS776HDNPA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.778277129 (515h53m49.649360174s) [noise=true , address=145.79.227.37:4566 , piece_id=PVEWMPWYO5HYKA7U3PVI5UGNYYM7N6SS433GXPZ2E6P3PRHQC4RA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.784658810 (515h53m49.655741855s) [noise=true , address=145.79.227.51:4409 , piece_id=X7OCV2VP23NYZ7DJIQFLN4AHHFHV5QYVPSUSXKJVPGGCQHGWAH5A, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.936874344 (515h53m49.807957389s) [noise=true , address=145.79.227.13:4804 , piece_id=DRBWXBP2FACIDODKAKRTXH7HVXCVXNLPOLZOMAZAUKSK4YIIQ3IA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:50.017222710 (515h53m49.888305755s) [noise=true , address=45.91.116.7:4012 , piece_id=RP6YNLP3ZEWJCFLCPVHHXAVRNF2Q52X4SAZGT7JWMAPFDL4VXEGQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:50.071741354 (515h53m49.942824399s) [noise=true , address=45.91.116.5:4300 , piece_id=GFLPWFM3LAG3LZ6VDPOIYHS7EMAET47YIAQESCV4CD66S7WZQEKA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:50.092145176 (515h53m49.963228211s) [noise=true , address=77.90.31.14:4663 , piece_id=B654ECCQPPQP37J534B2NT2BZIWO74QRFP3W22V3JMQFKDEFQYIA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:50.094428313 (515h53m49.965511348s) [noise=true , address=77.90.31.19:4844 , piece_id=TGJXM437WDPCCPFTCWP342LRB4A4R4HVW7PFZRVLGAB2RC2GX6RQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:50.094440367 (515h53m49.965523402s) [seg_pos={0 0}, upload=452664]: Segment reached optimal threshold of 13 pieces. Dec 27 03:22:50.094671677 (515h53m49.965754722s) [noise=true , address=145.79.227.33:4870 , piece_id=T43HLNGMWRYV3DXIIN3MAW2P6JNXKRHG4UNXK3EVYO2332VUROXA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 03:22:50.094682458 (515h53m49.965765493s) [noise=true , address=145.79.227.33:4870 , piece_id=T43HLNGMWRYV3DXIIN3MAW2P6JNXKRHG4UNXK3EVYO2332VUROXA, seg_pos={0 0}, upload=452664]: Long tail cancellation due to optimal threshold Dec 27 03:22:50.094782248 (515h53m49.965865283s) [noise=true , address=145.79.227.5:4222 , piece_id=DIEYHW5PTSPHVBS7FZRUJ5ATLFX45S7ZABVPYOUDTDA7NBKZTGSQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:59378->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:59378->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59378->145.79.227.5:4222: use of closed network connection Dec 27 03:22:50.094790353 (515h53m49.965873388s) [noise=true , address=145.79.227.5:4222 , piece_id=DIEYHW5PTSPHVBS7FZRUJ5ATLFX45S7ZABVPYOUDTDA7NBKZTGSQ, seg_pos={0 0}, upload=452664]: Long tail cancellation due to optimal threshold Dec 27 03:22:50.094842722 (515h53m49.965925757s) [noise=true , address=77.90.31.15:4700 , piece_id=NPPPGRW6GX2RW4QW7HJGQBKWIPJF62M3ANQ2453VD47N6EIOZ4BA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:48702->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:48702->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48702->77.90.31.15:4700: use of closed network connection Dec 27 03:22:50.094848725 (515h53m49.96593176s) [noise=true , address=77.90.31.15:4700 , piece_id=NPPPGRW6GX2RW4QW7HJGQBKWIPJF62M3ANQ2453VD47N6EIOZ4BA, seg_pos={0 0}, upload=452664]: Long tail cancellation due to optimal threshold Dec 27 03:22:50.094913247 (515h53m49.965996292s) [seg_pos={0 0}, upload=452664]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:22:50.094998129 (515h53m49.966081164s) [upload=452664]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:22:50.095002767 (515h53m49.966085803s) [upload=452664]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:22:50.398825778 (515h53m50.269908833s) [upload=452664]: Done uploading segments... Dec 27 03:22:50.398853391 (515h53m50.269936436s) [upload=452664]: Upload finished. err: <nil> Duration: 6.123659383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 22.947541861s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:23:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 22.947558855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XX.XXX.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.X] duration: 30.002859717s trace: https://jaeger.dev.storj.io/trace/525d751e7b0968da FAIL (2025-12-27 03:23:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GFLPWFM3LAG3LZ6VDPOIYHS7EMAET47YIAQESCV4CD66S7WZQEKA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: B654ECCQPPQP37J534B2NT2BZIWO74QRFP3W22V3JMQFKDEFQYIA) context deadline exceeded ======LOGS====== segments... Dec 27 03:22:47.491017930 (515h53m47.362100975s) [upload=452664]: Got next segment. Inline: false Dec 27 03:22:47.491065400 (515h53m47.362148465s) [upload=452664]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:22:47.491071713 (515h53m47.362154798s) [upload=452664]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:22:47.737055567 (515h53m47.608138602s) [seg_pos={0 0}, upload=452664]: Begin upload segment... Dec 27 03:22:47.737106595 (515h53m47.60818963s) [seg_pos={0 0}, upload=452664]: Done begin upload segment. Dec 27 03:22:47.737146771 (515h53m47.608229816s) [noise=true , address=45.91.116.7:4012 , piece_id=RP6YNLP3ZEWJCFLCPVHHXAVRNF2Q52X4SAZGT7JWMAPFDL4VXEGQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.737221824 (515h53m47.608304859s) [noise=true , address=145.79.227.13:4804 , piece_id=DRBWXBP2FACIDODKAKRTXH7HVXCVXNLPOLZOMAZAUKSK4YIIQ3IA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.737763656 (515h53m47.608846691s) [noise=true , address=145.79.227.38:4979 , piece_id=YWLB24X64WWO7ZSTXCKXH2IPMCXG3ZRDXSONNI4CCG47X3ESCATQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.738206840 (515h53m47.609289875s) [noise=true , address=145.79.227.45:4229 , piece_id=2BQBVCWRJGHNA3GMTQGJDIFLQ33JXDVEQGOOOIBD4EMGOEFI27AQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.738584458 (515h53m47.609667494s) [noise=true , address=145.79.227.33:4870 , piece_id=T43HLNGMWRYV3DXIIN3MAW2P6JNXKRHG4UNXK3EVYO2332VUROXA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.738873550 (515h53m47.609956585s) [noise=true , address=145.79.227.48:4303 , piece_id=GPDMVO2REPZTQHCEKFRJ2GFPJIML6OJREW6I5UBM53YLHPMJTKLQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.739143373 (515h53m47.610226408s) [noise=true , address=45.91.116.5:4300 , piece_id=GFLPWFM3LAG3LZ6VDPOIYHS7EMAET47YIAQESCV4CD66S7WZQEKA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.739504661 (515h53m47.610587706s) [noise=true , address=77.90.31.19:4844 , piece_id=TGJXM437WDPCCPFTCWP342LRB4A4R4HVW7PFZRVLGAB2RC2GX6RQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.740019492 (515h53m47.611102537s) [noise=true , address=77.90.31.14:4663 , piece_id=B654ECCQPPQP37J534B2NT2BZIWO74QRFP3W22V3JMQFKDEFQYIA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.740832980 (515h53m47.611916016s) [noise=true , address=145.79.227.5:4222 , piece_id=DIEYHW5PTSPHVBS7FZRUJ5ATLFX45S7ZABVPYOUDTDA7NBKZTGSQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.742032795 (515h53m47.61311583s) [noise=true , address=145.79.227.15:4874 , piece_id=I2XQWSAKEXC5HK273DSQVUMY5MBRKH33YDKMZ6ZWPQHZHWZ5KANQ, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.743113472 (515h53m47.614196517s) [noise=true , address=145.79.227.9:4365 , piece_id=5DLCWGLV72EETRYAXBY2Z7A3WBYZHBGFHENUPXAQWBSS776HDNPA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.744484213 (515h53m47.615567248s) [noise=true , address=145.79.227.37:4566 , piece_id=PVEWMPWYO5HYKA7U3PVI5UGNYYM7N6SS433GXPZ2E6P3PRHQC4RA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.746082677 (515h53m47.617165712s) [noise=true , address=145.79.227.51:4409 , piece_id=X7OCV2VP23NYZ7DJIQFLN4AHHFHV5QYVPSUSXKJVPGGCQHGWAH5A, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.749012185 (515h53m47.62009523s) [noise=true , address=77.90.31.15:4700 , piece_id=NPPPGRW6GX2RW4QW7HJGQBKWIPJF62M3ANQ2453VD47N6EIOZ4BA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:47.752041073 (515h53m47.623124118s) [noise=true , address=145.79.227.8:4330 , piece_id=GIK2VOJHRCTLGRXAMPCCQTB4BUUR465RVFQXQ5BB4D24FWQHYNRA, seg_pos={0 0}, upload=452664]: Uploading piece... Dec 27 03:22:49.443921459 (515h53m49.315004494s) [upload=452664]: Next returned nil segment Dec 27 03:22:49.443935365 (515h53m49.3150184s) [upload=452664]: Waiting for error group managing segments... Dec 27 03:22:49.737767541 (515h53m49.608850586s) [noise=true , address=145.79.227.48:4303 , piece_id=GPDMVO2REPZTQHCEKFRJ2GFPJIML6OJREW6I5UBM53YLHPMJTKLQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.738562064 (515h53m49.609645109s) [noise=true , address=145.79.227.8:4330 , piece_id=GIK2VOJHRCTLGRXAMPCCQTB4BUUR465RVFQXQ5BB4D24FWQHYNRA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.754288721 (515h53m49.625371756s) [noise=true , address=145.79.227.15:4874 , piece_id=I2XQWSAKEXC5HK273DSQVUMY5MBRKH33YDKMZ6ZWPQHZHWZ5KANQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.755430825 (515h53m49.62651387s) [noise=true , address=145.79.227.45:4229 , piece_id=2BQBVCWRJGHNA3GMTQGJDIFLQ33JXDVEQGOOOIBD4EMGOEFI27AQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.762945314 (515h53m49.634028349s) [noise=true , address=145.79.227.38:4979 , piece_id=YWLB24X64WWO7ZSTXCKXH2IPMCXG3ZRDXSONNI4CCG47X3ESCATQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.766075334 (515h53m49.637158379s) [noise=true , address=145.79.227.9:4365 , piece_id=5DLCWGLV72EETRYAXBY2Z7A3WBYZHBGFHENUPXAQWBSS776HDNPA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.778277129 (515h53m49.649360174s) [noise=true , address=145.79.227.37:4566 , piece_id=PVEWMPWYO5HYKA7U3PVI5UGNYYM7N6SS433GXPZ2E6P3PRHQC4RA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.784658810 (515h53m49.655741855s) [noise=true , address=145.79.227.51:4409 , piece_id=X7OCV2VP23NYZ7DJIQFLN4AHHFHV5QYVPSUSXKJVPGGCQHGWAH5A, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:49.936874344 (515h53m49.807957389s) [noise=true , address=145.79.227.13:4804 , piece_id=DRBWXBP2FACIDODKAKRTXH7HVXCVXNLPOLZOMAZAUKSK4YIIQ3IA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:50.017222710 (515h53m49.888305755s) [noise=true , address=45.91.116.7:4012 , piece_id=RP6YNLP3ZEWJCFLCPVHHXAVRNF2Q52X4SAZGT7JWMAPFDL4VXEGQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:50.071741354 (515h53m49.942824399s) [noise=true , address=45.91.116.5:4300 , piece_id=GFLPWFM3LAG3LZ6VDPOIYHS7EMAET47YIAQESCV4CD66S7WZQEKA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:50.092145176 (515h53m49.963228211s) [noise=true , address=77.90.31.14:4663 , piece_id=B654ECCQPPQP37J534B2NT2BZIWO74QRFP3W22V3JMQFKDEFQYIA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:50.094428313 (515h53m49.965511348s) [noise=true , address=77.90.31.19:4844 , piece_id=TGJXM437WDPCCPFTCWP342LRB4A4R4HVW7PFZRVLGAB2RC2GX6RQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: <nil> Dec 27 03:22:50.094440367 (515h53m49.965523402s) [seg_pos={0 0}, upload=452664]: Segment reached optimal threshold of 13 pieces. Dec 27 03:22:50.094671677 (515h53m49.965754722s) [noise=true , address=145.79.227.33:4870 , piece_id=T43HLNGMWRYV3DXIIN3MAW2P6JNXKRHG4UNXK3EVYO2332VUROXA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 03:22:50.094682458 (515h53m49.965765493s) [noise=true , address=145.79.227.33:4870 , piece_id=T43HLNGMWRYV3DXIIN3MAW2P6JNXKRHG4UNXK3EVYO2332VUROXA, seg_pos={0 0}, upload=452664]: Long tail cancellation due to optimal threshold Dec 27 03:22:50.094782248 (515h53m49.965865283s) [noise=true , address=145.79.227.5:4222 , piece_id=DIEYHW5PTSPHVBS7FZRUJ5ATLFX45S7ZABVPYOUDTDA7NBKZTGSQ, seg_pos={0 0}, upload=452664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:59378->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:59378->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59378->145.79.227.5:4222: use of closed network connection Dec 27 03:22:50.094790353 (515h53m49.965873388s) [noise=true , address=145.79.227.5:4222 , piece_id=DIEYHW5PTSPHVBS7FZRUJ5ATLFX45S7ZABVPYOUDTDA7NBKZTGSQ, seg_pos={0 0}, upload=452664]: Long tail cancellation due to optimal threshold Dec 27 03:22:50.094842722 (515h53m49.965925757s) [noise=true , address=77.90.31.15:4700 , piece_id=NPPPGRW6GX2RW4QW7HJGQBKWIPJF62M3ANQ2453VD47N6EIOZ4BA, seg_pos={0 0}, upload=452664]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:48702->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:48702->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48702->77.90.31.15:4700: use of closed network connection Dec 27 03:22:50.094848725 (515h53m49.96593176s) [noise=true , address=77.90.31.15:4700 , piece_id=NPPPGRW6GX2RW4QW7HJGQBKWIPJF62M3ANQ2453VD47N6EIOZ4BA, seg_pos={0 0}, upload=452664]: Long tail cancellation due to optimal threshold Dec 27 03:22:50.094913247 (515h53m49.965996292s) [seg_pos={0 0}, upload=452664]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:22:50.094998129 (515h53m49.966081164s) [upload=452664]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:22:50.095002767 (515h53m49.966085803s) [upload=452664]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:22:50.398825778 (515h53m50.269908833s) [upload=452664]: Done uploading segments... Dec 27 03:22:50.398853391 (515h53m50.269936436s) [upload=452664]: Upload finished. err: <nil> Duration: 30.002883803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 48.67165149s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:23:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 48.671668943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [..XX.XXX.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X.] duration: 6.178200957s trace: https://jaeger.dev.storj.io/trace/7a18012087fa9847 FAIL (2025-12-27 03:22:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: FYL6WSCWRJSTAPBEFZM3RNE7KY6OHZ4FDNANAXE2QJKW7JTWAXTQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: G6PLXPSKDGD7LJGFM3R6YZFKJPG6NZNZMKP4EC3JGTQQVNM4QPGA) context deadline exceeded ======LOGS====== Dec 27 03:21:15.445982458 (515h52m15.317065493s) [upload=452642]: Starting upload Dec 27 03:21:15.446150307 (515h52m15.317233382s) [upload=452642]: Done starting upload Dec 27 03:21:15.446269204 (515h52m15.317352249s) [upload=452642]: Uploading segments... Dec 27 03:21:15.446407467 (515h52m15.317490502s) [upload=452642]: Got next segment. Inline: false Dec 27 03:21:15.446661370 (515h52m15.317744405s) [upload=452642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:21:15.446670597 (515h52m15.317753642s) [upload=452642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:21:15.692393297 (515h52m15.563476342s) [seg_pos={0 0}, upload=452642]: Begin upload segment... Dec 27 03:21:15.692438643 (515h52m15.563521678s) [seg_pos={0 0}, upload=452642]: Done begin upload segment. Dec 27 03:21:15.692456176 (515h52m15.563539211s) [noise=true , address=145.79.227.47:4267 , piece_id=OKSRDPIHRWDQ46Z7KA4DYKL4WGREZVVI4TG4BA7GJSF6RFY2TRDQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.692469512 (515h52m15.563552557s) [noise=true , address=145.79.227.50:4375 , piece_id=2Y5RADELA2GJJ5K3JY3PXXINR3LR4ZUOVIMYEWYGHOFQW32MCKVQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.692922815 (515h52m15.56400585s) [noise=true , address=45.91.116.12:4227 , piece_id=WFGOI3GSKCLZUWZ66O5DHNXG7DVS45EQ22WCEKCJWNBB37MBKCEQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.693019509 (515h52m15.564102544s) [noise=true , address=145.79.227.7:4293 , piece_id=JUBDPWXHP2HW3DRHPX3DVTAXSDFIKAAQWKOTJQ5WXJ2FKWIKSEGA, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.693367212 (515h52m15.564450277s) [noise=true , address=145.79.227.24:4582 , piece_id=ZEAR74KIJJAZWAO5V2A63CLG7VA3NKBKQEQHSSU33XKGZYLV2EZQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.694014714 (515h52m15.565097749s) [noise=true , address=145.79.227.3:4149 , piece_id=BM2OJIIST6BO7DRTE25SL6FYNOYHXEDKPSQM3SUSBOVT3GJTGNCQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.694598957 (515h52m15.565681992s) [noise=true , address=145.79.227.44:4193 , piece_id=IGKXX3K5BSX4UXAQAF7RC5BDQNYDK6EIGYXF3JLXNOO6CMOUV52Q, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.694947370 (515h52m15.566030415s) [noise=true , address=145.79.227.37:4566 , piece_id=TJYVTCCRDWWPHO7WIZQJED2GXHNJTGB2N5C5GDMWXG6ULLY2C2TQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.695011643 (515h52m15.566094688s) [noise=true , address=145.79.227.41:4085 , piece_id=7LU2TTC6IIZXVXBXIRU34DIZFYQJF5EYPAPQR3KXNPEATAYFEJ3Q, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.695306844 (515h52m15.566389889s) [noise=true , address=145.79.227.16:4912 , piece_id=MPRQUQ7DOATVTC3HKRPQLFUZN3THEURLUCIDATQS5YRQ5DYEYHOA, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.695345448 (515h52m15.566428483s) [noise=true , address=45.91.116.13:4155 , piece_id=G6PLXPSKDGD7LJGFM3R6YZFKJPG6NZNZMKP4EC3JGTQQVNM4QPGA, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.695564596 (515h52m15.566647631s) [noise=true , address=145.79.227.8:4330 , piece_id=7PODDMST4OIQ4NWZBGPHK6JCC3XJL76H3ENWOMBKEBN7LCERU6CQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.695815463 (515h52m15.566898498s) [noise=true , address=145.79.227.42:4122 , piece_id=5J5H6PWYUAHOZNFOH4MI4TGNMLLYWMLQ2FVZW5RMOUX57JEXJRGA, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.696042234 (515h52m15.567125269s) [noise=true , address=45.91.116.6:4973 , piece_id=NDFWQAENEPLHT5Q5VJZILMOQSOHKHU4565VEQB5MVRYURMNORT4Q, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.696289676 (515h52m15.567372711s) [noise=true , address=145.79.227.46:4603 , piece_id=XSONSARWKLO5P7YQOA3TJKPSFBUZQ7QNGHI272ZQKFQKDTMKYPGA, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.696543920 (515h52m15.567626965s) [noise=true , address=45.91.116.14:4190 , piece_id=FYL6WSCWRJSTAPBEFZM3RNE7KY6OHZ4FDNANAXE2QJKW7JTWAXTQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:17.314065837 (515h52m17.185148882s) [upload=452642]: Next returned nil segment Dec 27 03:21:17.314078861 (515h52m17.185161897s) [upload=452642]: Waiting for error group managing segments... Dec 27 03:21:17.695348143 (515h52m17.566431188s) [noise=true , address=145.79.227.7:4293 , piece_id=JUBDPWXHP2HW3DRHPX3DVTAXSDFIKAAQWKOTJQ5WXJ2FKWIKSEGA, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.700656551 (515h52m17.571739586s) [noise=true , address=145.79.227.41:4085 , piece_id=7LU2TTC6IIZXVXBXIRU34DIZFYQJF5EYPAPQR3KXNPEATAYFEJ3Q, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.701499226 (515h52m17.572582271s) [noise=true , address=145.79.227.37:4566 , piece_id=TJYVTCCRDWWPHO7WIZQJED2GXHNJTGB2N5C5GDMWXG6ULLY2C2TQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.703250932 (515h52m17.574333967s) [noise=true , address=145.79.227.16:4912 , piece_id=MPRQUQ7DOATVTC3HKRPQLFUZN3THEURLUCIDATQS5YRQ5DYEYHOA, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.709190722 (515h52m17.580273757s) [noise=true , address=145.79.227.24:4582 , piece_id=ZEAR74KIJJAZWAO5V2A63CLG7VA3NKBKQEQHSSU33XKGZYLV2EZQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.709775555 (515h52m17.5808586s) [noise=true , address=145.79.227.46:4603 , piece_id=XSONSARWKLO5P7YQOA3TJKPSFBUZQ7QNGHI272ZQKFQKDTMKYPGA, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.715532398 (515h52m17.586615443s) [noise=true , address=145.79.227.50:4375 , piece_id=2Y5RADELA2GJJ5K3JY3PXXINR3LR4ZUOVIMYEWYGHOFQW32MCKVQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.723193416 (515h52m17.594276461s) [noise=true , address=145.79.227.42:4122 , piece_id=5J5H6PWYUAHOZNFOH4MI4TGNMLLYWMLQ2FVZW5RMOUX57JEXJRGA, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.725464580 (515h52m17.596547625s) [noise=true , address=145.79.227.8:4330 , piece_id=7PODDMST4OIQ4NWZBGPHK6JCC3XJL76H3ENWOMBKEBN7LCERU6CQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.730642931 (515h52m17.601725976s) [noise=true , address=145.79.227.47:4267 , piece_id=OKSRDPIHRWDQ46Z7KA4DYKL4WGREZVVI4TG4BA7GJSF6RFY2TRDQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.932524557 (515h52m17.803607592s) [noise=true , address=45.91.116.13:4155 , piece_id=G6PLXPSKDGD7LJGFM3R6YZFKJPG6NZNZMKP4EC3JGTQQVNM4QPGA, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.967605928 (515h52m17.838688973s) [noise=true , address=45.91.116.12:4227 , piece_id=WFGOI3GSKCLZUWZ66O5DHNXG7DVS45EQ22WCEKCJWNBB37MBKCEQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.985383791 (515h52m17.856466836s) [noise=true , address=45.91.116.14:4190 , piece_id=FYL6WSCWRJSTAPBEFZM3RNE7KY6OHZ4FDNANAXE2QJKW7JTWAXTQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.985409680 (515h52m17.856492725s) [seg_pos={0 0}, upload=452642]: Segment reached optimal threshold of 13 pieces. Dec 27 03:21:17.985582468 (515h52m17.856665513s) [noise=true , address=45.91.116.6:4973 , piece_id=NDFWQAENEPLHT5Q5VJZILMOQSOHKHU4565VEQB5MVRYURMNORT4Q, seg_pos={0 0}, upload=452642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 27 03:21:17.985588811 (515h52m17.856671856s) [noise=true , address=45.91.116.6:4973 , piece_id=NDFWQAENEPLHT5Q5VJZILMOQSOHKHU4565VEQB5MVRYURMNORT4Q, seg_pos={0 0}, upload=452642]: Long tail cancellation due to optimal threshold Dec 27 03:21:17.985870187 (515h52m17.856953222s) [noise=true , address=145.79.227.3:4149 , piece_id=BM2OJIIST6BO7DRTE25SL6FYNOYHXEDKPSQM3SUSBOVT3GJTGNCQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Dec 27 03:21:17.985877350 (515h52m17.856960395s) [noise=true , address=145.79.227.3:4149 , piece_id=BM2OJIIST6BO7DRTE25SL6FYNOYHXEDKPSQM3SUSBOVT3GJTGNCQ, seg_pos={0 0}, upload=452642]: Long tail cancellation due to optimal threshold Dec 27 03:21:17.985994333 (515h52m17.857077368s) [noise=true , address=145.79.227.44:4193 , piece_id=IGKXX3K5BSX4UXAQAF7RC5BDQNYDK6EIGYXF3JLXNOO6CMOUV52Q, seg_pos={0 0}, upload=452642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 03:21:17.986000414 (515h52m17.857083449s) [noise=true , address=145.79.227.44:4193 , piece_id=IGKXX3K5BSX4UXAQAF7RC5BDQNYDK6EIGYXF3JLXNOO6CMOUV52Q, seg_pos={0 0}, upload=452642]: Long tail cancellation due to optimal threshold Dec 27 03:21:17.986016655 (515h52m17.857099701s) [seg_pos={0 0}, upload=452642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:21:17.986079465 (515h52m17.8571625s) [upload=452642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:21:17.986084765 (515h52m17.8571678s) [upload=452642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:21:18.281875190 (515h52m18.152958235s) [upload=452642]: Done uploading segments... Dec 27 03:21:18.281900569 (515h52m18.152983624s) [upload=452642]: Upload finished. err: <nil> Duration: 6.1782342s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 51.300814187s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:22:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 51.30091069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [...XX.XXX.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.X] duration: 30.003162684s trace: https://jaeger.dev.storj.io/trace/7a18012087fa9847 FAIL (2025-12-27 03:21:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: FYL6WSCWRJSTAPBEFZM3RNE7KY6OHZ4FDNANAXE2QJKW7JTWAXTQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: G6PLXPSKDGD7LJGFM3R6YZFKJPG6NZNZMKP4EC3JGTQQVNM4QPGA) context deadline exceeded ======LOGS====== Dec 27 03:21:15.445982458 (515h52m15.317065493s) [upload=452642]: Starting upload Dec 27 03:21:15.446150307 (515h52m15.317233382s) [upload=452642]: Done starting upload Dec 27 03:21:15.446269204 (515h52m15.317352249s) [upload=452642]: Uploading segments... Dec 27 03:21:15.446407467 (515h52m15.317490502s) [upload=452642]: Got next segment. Inline: false Dec 27 03:21:15.446661370 (515h52m15.317744405s) [upload=452642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:21:15.446670597 (515h52m15.317753642s) [upload=452642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:21:15.692393297 (515h52m15.563476342s) [seg_pos={0 0}, upload=452642]: Begin upload segment... Dec 27 03:21:15.692438643 (515h52m15.563521678s) [seg_pos={0 0}, upload=452642]: Done begin upload segment. Dec 27 03:21:15.692456176 (515h52m15.563539211s) [noise=true , address=145.79.227.47:4267 , piece_id=OKSRDPIHRWDQ46Z7KA4DYKL4WGREZVVI4TG4BA7GJSF6RFY2TRDQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.692469512 (515h52m15.563552557s) [noise=true , address=145.79.227.50:4375 , piece_id=2Y5RADELA2GJJ5K3JY3PXXINR3LR4ZUOVIMYEWYGHOFQW32MCKVQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.692922815 (515h52m15.56400585s) [noise=true , address=45.91.116.12:4227 , piece_id=WFGOI3GSKCLZUWZ66O5DHNXG7DVS45EQ22WCEKCJWNBB37MBKCEQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.693019509 (515h52m15.564102544s) [noise=true , address=145.79.227.7:4293 , piece_id=JUBDPWXHP2HW3DRHPX3DVTAXSDFIKAAQWKOTJQ5WXJ2FKWIKSEGA, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.693367212 (515h52m15.564450277s) [noise=true , address=145.79.227.24:4582 , piece_id=ZEAR74KIJJAZWAO5V2A63CLG7VA3NKBKQEQHSSU33XKGZYLV2EZQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.694014714 (515h52m15.565097749s) [noise=true , address=145.79.227.3:4149 , piece_id=BM2OJIIST6BO7DRTE25SL6FYNOYHXEDKPSQM3SUSBOVT3GJTGNCQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.694598957 (515h52m15.565681992s) [noise=true , address=145.79.227.44:4193 , piece_id=IGKXX3K5BSX4UXAQAF7RC5BDQNYDK6EIGYXF3JLXNOO6CMOUV52Q, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.694947370 (515h52m15.566030415s) [noise=true , address=145.79.227.37:4566 , piece_id=TJYVTCCRDWWPHO7WIZQJED2GXHNJTGB2N5C5GDMWXG6ULLY2C2TQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.695011643 (515h52m15.566094688s) [noise=true , address=145.79.227.41:4085 , piece_id=7LU2TTC6IIZXVXBXIRU34DIZFYQJF5EYPAPQR3KXNPEATAYFEJ3Q, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.695306844 (515h52m15.566389889s) [noise=true , address=145.79.227.16:4912 , piece_id=MPRQUQ7DOATVTC3HKRPQLFUZN3THEURLUCIDATQS5YRQ5DYEYHOA, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.695345448 (515h52m15.566428483s) [noise=true , address=45.91.116.13:4155 , piece_id=G6PLXPSKDGD7LJGFM3R6YZFKJPG6NZNZMKP4EC3JGTQQVNM4QPGA, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.695564596 (515h52m15.566647631s) [noise=true , address=145.79.227.8:4330 , piece_id=7PODDMST4OIQ4NWZBGPHK6JCC3XJL76H3ENWOMBKEBN7LCERU6CQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.695815463 (515h52m15.566898498s) [noise=true , address=145.79.227.42:4122 , piece_id=5J5H6PWYUAHOZNFOH4MI4TGNMLLYWMLQ2FVZW5RMOUX57JEXJRGA, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.696042234 (515h52m15.567125269s) [noise=true , address=45.91.116.6:4973 , piece_id=NDFWQAENEPLHT5Q5VJZILMOQSOHKHU4565VEQB5MVRYURMNORT4Q, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.696289676 (515h52m15.567372711s) [noise=true , address=145.79.227.46:4603 , piece_id=XSONSARWKLO5P7YQOA3TJKPSFBUZQ7QNGHI272ZQKFQKDTMKYPGA, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:15.696543920 (515h52m15.567626965s) [noise=true , address=45.91.116.14:4190 , piece_id=FYL6WSCWRJSTAPBEFZM3RNE7KY6OHZ4FDNANAXE2QJKW7JTWAXTQ, seg_pos={0 0}, upload=452642]: Uploading piece... Dec 27 03:21:17.314065837 (515h52m17.185148882s) [upload=452642]: Next returned nil segment Dec 27 03:21:17.314078861 (515h52m17.185161897s) [upload=452642]: Waiting for error group managing segments... Dec 27 03:21:17.695348143 (515h52m17.566431188s) [noise=true , address=145.79.227.7:4293 , piece_id=JUBDPWXHP2HW3DRHPX3DVTAXSDFIKAAQWKOTJQ5WXJ2FKWIKSEGA, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.700656551 (515h52m17.571739586s) [noise=true , address=145.79.227.41:4085 , piece_id=7LU2TTC6IIZXVXBXIRU34DIZFYQJF5EYPAPQR3KXNPEATAYFEJ3Q, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.701499226 (515h52m17.572582271s) [noise=true , address=145.79.227.37:4566 , piece_id=TJYVTCCRDWWPHO7WIZQJED2GXHNJTGB2N5C5GDMWXG6ULLY2C2TQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.703250932 (515h52m17.574333967s) [noise=true , address=145.79.227.16:4912 , piece_id=MPRQUQ7DOATVTC3HKRPQLFUZN3THEURLUCIDATQS5YRQ5DYEYHOA, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.709190722 (515h52m17.580273757s) [noise=true , address=145.79.227.24:4582 , piece_id=ZEAR74KIJJAZWAO5V2A63CLG7VA3NKBKQEQHSSU33XKGZYLV2EZQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.709775555 (515h52m17.5808586s) [noise=true , address=145.79.227.46:4603 , piece_id=XSONSARWKLO5P7YQOA3TJKPSFBUZQ7QNGHI272ZQKFQKDTMKYPGA, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.715532398 (515h52m17.586615443s) [noise=true , address=145.79.227.50:4375 , piece_id=2Y5RADELA2GJJ5K3JY3PXXINR3LR4ZUOVIMYEWYGHOFQW32MCKVQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.723193416 (515h52m17.594276461s) [noise=true , address=145.79.227.42:4122 , piece_id=5J5H6PWYUAHOZNFOH4MI4TGNMLLYWMLQ2FVZW5RMOUX57JEXJRGA, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.725464580 (515h52m17.596547625s) [noise=true , address=145.79.227.8:4330 , piece_id=7PODDMST4OIQ4NWZBGPHK6JCC3XJL76H3ENWOMBKEBN7LCERU6CQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.730642931 (515h52m17.601725976s) [noise=true , address=145.79.227.47:4267 , piece_id=OKSRDPIHRWDQ46Z7KA4DYKL4WGREZVVI4TG4BA7GJSF6RFY2TRDQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.932524557 (515h52m17.803607592s) [noise=true , address=45.91.116.13:4155 , piece_id=G6PLXPSKDGD7LJGFM3R6YZFKJPG6NZNZMKP4EC3JGTQQVNM4QPGA, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.967605928 (515h52m17.838688973s) [noise=true , address=45.91.116.12:4227 , piece_id=WFGOI3GSKCLZUWZ66O5DHNXG7DVS45EQ22WCEKCJWNBB37MBKCEQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.985383791 (515h52m17.856466836s) [noise=true , address=45.91.116.14:4190 , piece_id=FYL6WSCWRJSTAPBEFZM3RNE7KY6OHZ4FDNANAXE2QJKW7JTWAXTQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: <nil> Dec 27 03:21:17.985409680 (515h52m17.856492725s) [seg_pos={0 0}, upload=452642]: Segment reached optimal threshold of 13 pieces. Dec 27 03:21:17.985582468 (515h52m17.856665513s) [noise=true , address=45.91.116.6:4973 , piece_id=NDFWQAENEPLHT5Q5VJZILMOQSOHKHU4565VEQB5MVRYURMNORT4Q, seg_pos={0 0}, upload=452642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 27 03:21:17.985588811 (515h52m17.856671856s) [noise=true , address=45.91.116.6:4973 , piece_id=NDFWQAENEPLHT5Q5VJZILMOQSOHKHU4565VEQB5MVRYURMNORT4Q, seg_pos={0 0}, upload=452642]: Long tail cancellation due to optimal threshold Dec 27 03:21:17.985870187 (515h52m17.856953222s) [noise=true , address=145.79.227.3:4149 , piece_id=BM2OJIIST6BO7DRTE25SL6FYNOYHXEDKPSQM3SUSBOVT3GJTGNCQ, seg_pos={0 0}, upload=452642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Dec 27 03:21:17.985877350 (515h52m17.856960395s) [noise=true , address=145.79.227.3:4149 , piece_id=BM2OJIIST6BO7DRTE25SL6FYNOYHXEDKPSQM3SUSBOVT3GJTGNCQ, seg_pos={0 0}, upload=452642]: Long tail cancellation due to optimal threshold Dec 27 03:21:17.985994333 (515h52m17.857077368s) [noise=true , address=145.79.227.44:4193 , piece_id=IGKXX3K5BSX4UXAQAF7RC5BDQNYDK6EIGYXF3JLXNOO6CMOUV52Q, seg_pos={0 0}, upload=452642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 03:21:17.986000414 (515h52m17.857083449s) [noise=true , address=145.79.227.44:4193 , piece_id=IGKXX3K5BSX4UXAQAF7RC5BDQNYDK6EIGYXF3JLXNOO6CMOUV52Q, seg_pos={0 0}, upload=452642]: Long tail cancellation due to optimal threshold Dec 27 03:21:17.986016655 (515h52m17.857099701s) [seg_pos={0 0}, upload=452642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:21:17.986079465 (515h52m17.8571625s) [upload=452642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:21:17.986084765 (515h52m17.8571678s) [upload=452642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:21:18.281875190 (515h52m18.152958235s) [upload=452642]: Done uploading segments... Dec 27 03:21:18.281900569 (515h52m18.152983624s) [upload=452642]: Upload finished. err: <nil> Duration: 30.003186621s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 37.436550451s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:21:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 37.436567935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 55.936951514s trace: https://jaeger.dev.storj.io/trace/41f771beaff8015f FAIL (2025-12-27 03:20:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, Piece ID: RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ) context deadline exceeded; piecestore: (Node ID: 12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, Piece ID: FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA) context deadline exceeded; piecestore: (Node ID: 1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, Piece ID: GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA) context deadline exceeded ======LOGS====== . Dec 27 02:34:46.697235242 (515h5m46.568318277s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698006120 (515h5m46.569089165s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.698822214 (515h5m46.569905259s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699464066 (515h5m46.570547101s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.699946535 (515h5m46.57102958s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.700853632 (515h5m46.571936677s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.702726839 (515h5m46.573809884s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.703171436 (515h5m46.574254481s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.704980119 (515h5m46.576063164s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.707019443 (515h5m46.578102488s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.708255476 (515h5m46.579338521s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.722909610 (515h5m46.593992645s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.727160063 (515h5m46.598243108s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.730029197 (515h5m46.601112232s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.738016847 (515h5m46.609099882s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:46.740503112 (515h5m46.611586157s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Uploading piece... Dec 27 02:34:48.032294283 (515h5m47.903377318s) [upload=451963]: Next returned nil segment Dec 27 02:34:48.032309673 (515h5m47.903392708s) [upload=451963]: Waiting for error group managing segments... Dec 27 02:34:48.428011994 (515h5m48.299095029s) [noise=true , address=64.49.13.137:4496 , piece_id=U6NR3U624PCLQUNPDRGXPQ5O7K7SDQMNQ3D46CG7IGJA63S647HQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.580353228 (515h5m48.451436263s) [noise=true , address=145.79.227.16:4926 , piece_id=5HWL66CTR3COHCFGZDTR6X62ROEZXHR45ADARDTM3K2PVLRYRB7A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.585146565 (515h5m48.4562296s) [noise=true , address=145.79.227.21:4490 , piece_id=BAQN5OEED7LQR6SP7W4QJDIJU5FNNCB4ASZN6N7QZFX722AFB7XQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.591034226 (515h5m48.462117271s) [noise=true , address=145.79.227.36:4551 , piece_id=VZKPIXYCPOKFLR6RI2YTUOOJEJRMBMI2N6MWIRAYQYINB7MA25QA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.596107276 (515h5m48.467190321s) [noise=true , address=145.79.227.42:4149 , piece_id=TDLECIFEWSSRIRLEIW3VOFQWGVOYCT4QX5QQ4FTRNR2HT475CHVA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.599194074 (515h5m48.470277119s) [noise=true , address=145.79.227.51:4437 , piece_id=DF44WYQWNI3TWDI3RXUCCEVCI7B62SFZE5GTQ5YL5MPIVFJ3XH2A, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.711484783 (515h5m48.582567818s) [noise=true , address=145.79.227.29:4957 , piece_id=AVTR5Q5JTZ2IUYRP57ZBYMRY23Y35TSMYBMPRJMEMAVO7SEN2Q4Q, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.782091410 (515h5m48.653174445s) [noise=true , address=77.90.31.16:4752 , piece_id=WM25FQ5H5PR7B3RPS372XNC2JUTOXXSESK46HMCICHKBKUDSO4IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.796068835 (515h5m48.66715187s) [noise=true , address=77.90.31.19:4860 , piece_id=BP3T6473MYS6KG2CAVNJ4AXMCSHHQ5JS67JT2AZSTI2F523FM5IA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.920348214 (515h5m48.791431249s) [noise=true , address=45.91.116.4:4959 , piece_id=B3OBKBV5POGCUY6NH7URZD4RYII7NSAAD7K4LJKNLDZ4LDGRSELA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.934488639 (515h5m48.805571684s) [noise=true , address=45.91.116.12:4256 , piece_id=E5NSYGJVDMHA773GURKZTORPBOV7ZLSSNISAGCGDDEA2AGHDXEKQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:48.942261590 (515h5m48.813344636s) [noise=true , address=45.91.116.9:4268 , piece_id=GU76ANZOA6JVCC22FLAGJLGJBV5XGI7BZKERIV4GM5FLBY4HJADA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.211338236 (515h5m49.082421271s) [noise=true , address=45.91.116.14:4203 , piece_id=FSML4OPH6UDA4JKTJXY26OGFPPC3S27FJHRQTY3M6BLEKT2HS7DA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216036673 (515h5m49.087119708s) [noise=true , address=77.90.31.14:4667 , piece_id=RFWJYERHFH4K4WQDF7JBE334BLD7TH5EGIMUDDJT2SVGOARZQFJQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: <nil> Dec 27 02:34:49.216054516 (515h5m49.087137561s) [seg_pos={0 0}, upload=451963]: Segment reached optimal threshold of 14 pieces. Dec 27 02:34:49.216235682 (515h5m49.087318717s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216241543 (515h5m49.087324578s) [noise=true , address=193.149.164.143:4695 , piece_id=ZJR3BF37HJUSHHDP4CAIDK6P24XUNEYMEBPKQC7Z5QWZRRLKJPCA, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216362102 (515h5m49.087445137s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39702->45.91.116.8:4063: use of closed network connection Dec 27 02:34:49.216380237 (515h5m49.087463272s) [noise=true , address=45.91.116.8:4063 , piece_id=PMATA56XVAH6RNXB3O7TVJGXSYCSR6HWH4OOK4ZZFODIFB7WTCWQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216404553 (515h5m49.087487598s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216419231 (515h5m49.087502276s) [noise=true , address=145.79.227.30:4796 , piece_id=USS7LF2R6IJC56S5AEE6PTYL2YWIY4EFCAPYFCWZMS5G25W6HBAQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216473614 (515h5m49.087556649s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz): protocol: expected piece hash; context canceled; EOF Dec 27 02:34:49.216479015 (515h5m49.08756205s) [noise=true , address=77.90.31.13:4631 , piece_id=AIFIEGAVA4XAYT2TMZQVHI6N7ONPPWRKFWIMT3ZVHENDGBBAZXUQ, seg_pos={0 0}, upload=451963]: Long tail cancellation due to optimal threshold Dec 27 02:34:49.216559619 (515h5m49.087642654s) [seg_pos={0 0}, upload=451963]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 02:34:49.216625574 (515h5m49.087708609s) [upload=451963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 02:34:49.216629782 (515h5m49.087712817s) [upload=451963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 02:34:49.504153845 (515h5m49.37523688s) [upload=451963]: Done uploading segments... Dec 27 02:34:49.504208278 (515h5m49.375291313s) [upload=451963]: Upload finished. err: <nil> Duration: 55.936967294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [....XX.XXX.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX.] duration: 5.712008228s trace: https://jaeger.dev.storj.io/trace/14e8898c1be08c07 FAIL (2025-12-27 03:20:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 26YTSAOTHXRYNX23JSUTV2EAHNAXHGRYJ5CDGZ2IJJVBJE6VJQMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: FTER3HR67PWQL2ZHJBOV7AKYGKJY5TFRROECUFDUJHWRREEF5ADQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: DOYCEVA3VWTXX3PEEN7RFYKNAQE5XJGXANSX5JNUQ2J34QFUQVMA) context deadline exceeded ======LOGS====== Dec 27 03:19:49.188903006 (515h50m49.059986071s) [upload=452621]: Starting upload Dec 27 03:19:49.188985262 (515h50m49.060068297s) [upload=452621]: Done starting upload Dec 27 03:19:49.189079181 (515h50m49.060162226s) [upload=452621]: Uploading segments... Dec 27 03:19:49.189279383 (515h50m49.060362428s) [upload=452621]: Got next segment. Inline: false Dec 27 03:19:49.189319980 (515h50m49.060403015s) [upload=452621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:19:49.189325210 (515h50m49.060408265s) [upload=452621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:19:49.434468927 (515h50m49.305551982s) [seg_pos={0 0}, upload=452621]: Begin upload segment... Dec 27 03:19:49.434586961 (515h50m49.305670006s) [seg_pos={0 0}, upload=452621]: Done begin upload segment. Dec 27 03:19:49.434622078 (515h50m49.305705133s) [noise=true , address=145.79.227.46:4602 , piece_id=UJDPKL63BUERHIBR7CXHOJF6X2P45LECNDCTH6JDNH3AEOUC45KA, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.434707291 (515h50m49.305790336s) [noise=true , address=45.91.116.9:4263 , piece_id=JANEC6VLCENGC4OEWNFRC23DQH4TFGS6HV5IYHRYQQA2UJ4YKZLA, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.435184499 (515h50m49.306267534s) [noise=true , address=145.79.227.16:4912 , piece_id=RBSEZV6CRPJTL3II5XC2T7RKOFXKF4ZLWY6S4YGFMHG7EORKJT3A, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.435598878 (515h50m49.306681913s) [noise=true , address=145.79.227.17:4970 , piece_id=TAA3JMA2NVNFL7BLDGW5LVFJROJX5GYEJOIMVQFGK3RRLBZZHAMQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.436016293 (515h50m49.307099328s) [noise=true , address=77.90.31.17:4770 , piece_id=INMMZWAQEFSWXNRNLPOMUSHVW4IXAEAM5JDV4PLLRJI6G4QALCQQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.436449328 (515h50m49.307532373s) [noise=true , address=45.91.116.14:4191 , piece_id=YU65LLS4ZE34XS66EQRFLARBBOZ7ZXO2VRQU7I3I6B47IEYEQTLQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.436867955 (515h50m49.307951s) [noise=true , address=45.91.116.8:4046 , piece_id=FTER3HR67PWQL2ZHJBOV7AKYGKJY5TFRROECUFDUJHWRREEF5ADQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.438103697 (515h50m49.309186732s) [noise=true , address=77.90.31.11:4554 , piece_id=6ZM7YLZIHPZXMK4D2XORBXPXN2WWWUE66PQPPJL3OVTZCIUATMPQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.438400583 (515h50m49.309483618s) [noise=true , address=145.79.227.9:4365 , piece_id=MLF3Q7XUPJP2O2AFOXDVT2GD6WFCDAOICO44LQ7FEF4WQKZOVL2Q, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.439157794 (515h50m49.310240829s) [noise=true , address=77.90.31.10:4518 , piece_id=DOYCEVA3VWTXX3PEEN7RFYKNAQE5XJGXANSX5JNUQ2J34QFUQVMA, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.439954161 (515h50m49.311037206s) [noise=true , address=145.79.227.45:4230 , piece_id=IR5WHYBFIJ3YIDK4FLI3WPUZYWIYI7FXOVVCQ4JETORYDHGEESTQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.440203115 (515h50m49.31128615s) [noise=true , address=45.91.116.6:4975 , piece_id=2PPMENMABXW2AUIRG6MVFKRMEPEZ5XNM2LIFD3VD7SPW4J3Z3CWQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.441209411 (515h50m49.312292446s) [noise=true , address=145.79.227.39:4013 , piece_id=237Q46IGU7OI3RQ7OLAEY5ZNW57UFTYCLUKSKGL3O6J2D4XJ3CVQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.442361094 (515h50m49.313444129s) [noise=true , address=145.79.227.37:4565 , piece_id=CD7HGSFMGDUWUUGRX4WX4ICU5VW7ARASTJERCMXE33YC4ZM7U7LA, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.443609141 (515h50m49.314692176s) [noise=true , address=45.91.116.12:4228 , piece_id=26YTSAOTHXRYNX23JSUTV2EAHNAXHGRYJ5CDGZ2IJJVBJE6VJQMA, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.443839039 (515h50m49.314922084s) [noise=true , address=145.79.227.35:4481 , piece_id=DMF5XLRTQHKOOFYXT2ATU4N3HCREMCWFF4ZQTQU4FWANR65Q5PYQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:51.162583489 (515h50m51.033666554s) [upload=452621]: Next returned nil segment Dec 27 03:19:51.162601392 (515h50m51.033684428s) [upload=452621]: Waiting for error group managing segments... Dec 27 03:19:51.440864473 (515h50m51.311947508s) [noise=true , address=145.79.227.45:4230 , piece_id=IR5WHYBFIJ3YIDK4FLI3WPUZYWIYI7FXOVVCQ4JETORYDHGEESTQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.446548336 (515h50m51.317631391s) [noise=true , address=145.79.227.37:4565 , piece_id=CD7HGSFMGDUWUUGRX4WX4ICU5VW7ARASTJERCMXE33YC4ZM7U7LA, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.452279288 (515h50m51.323362333s) [noise=true , address=145.79.227.39:4013 , piece_id=237Q46IGU7OI3RQ7OLAEY5ZNW57UFTYCLUKSKGL3O6J2D4XJ3CVQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.455079651 (515h50m51.326162696s) [noise=true , address=145.79.227.46:4602 , piece_id=UJDPKL63BUERHIBR7CXHOJF6X2P45LECNDCTH6JDNH3AEOUC45KA, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.455126359 (515h50m51.326209394s) [noise=true , address=145.79.227.17:4970 , piece_id=TAA3JMA2NVNFL7BLDGW5LVFJROJX5GYEJOIMVQFGK3RRLBZZHAMQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.466422419 (515h50m51.337505454s) [noise=true , address=145.79.227.9:4365 , piece_id=MLF3Q7XUPJP2O2AFOXDVT2GD6WFCDAOICO44LQ7FEF4WQKZOVL2Q, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.468365870 (515h50m51.339448905s) [noise=true , address=145.79.227.35:4481 , piece_id=DMF5XLRTQHKOOFYXT2ATU4N3HCREMCWFF4ZQTQU4FWANR65Q5PYQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.477709802 (515h50m51.348792847s) [noise=true , address=145.79.227.16:4912 , piece_id=RBSEZV6CRPJTL3II5XC2T7RKOFXKF4ZLWY6S4YGFMHG7EORKJT3A, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.623148793 (515h50m51.494231838s) [noise=true , address=77.90.31.11:4554 , piece_id=6ZM7YLZIHPZXMK4D2XORBXPXN2WWWUE66PQPPJL3OVTZCIUATMPQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.717006273 (515h50m51.588089318s) [noise=true , address=45.91.116.12:4228 , piece_id=26YTSAOTHXRYNX23JSUTV2EAHNAXHGRYJ5CDGZ2IJJVBJE6VJQMA, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.729552654 (515h50m51.600635689s) [noise=true , address=77.90.31.10:4518 , piece_id=DOYCEVA3VWTXX3PEEN7RFYKNAQE5XJGXANSX5JNUQ2J34QFUQVMA, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.814652328 (515h50m51.685735373s) [noise=true , address=45.91.116.8:4046 , piece_id=FTER3HR67PWQL2ZHJBOV7AKYGKJY5TFRROECUFDUJHWRREEF5ADQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:52.422414991 (515h50m52.293498026s) [noise=true , address=45.91.116.9:4263 , piece_id=JANEC6VLCENGC4OEWNFRC23DQH4TFGS6HV5IYHRYQQA2UJ4YKZLA, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:52.422433807 (515h50m52.293516842s) [seg_pos={0 0}, upload=452621]: Segment reached optimal threshold of 13 pieces. Dec 27 03:19:52.422647394 (515h50m52.293730429s) [noise=true , address=77.90.31.17:4770 , piece_id=INMMZWAQEFSWXNRNLPOMUSHVW4IXAEAM5JDV4PLLRJI6G4QALCQQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 03:19:52.422654206 (515h50m52.293737241s) [noise=true , address=77.90.31.17:4770 , piece_id=INMMZWAQEFSWXNRNLPOMUSHVW4IXAEAM5JDV4PLLRJI6G4QALCQQ, seg_pos={0 0}, upload=452621]: Long tail cancellation due to optimal threshold Dec 27 03:19:52.422756161 (515h50m52.293839196s) [noise=true , address=45.91.116.6:4975 , piece_id=2PPMENMABXW2AUIRG6MVFKRMEPEZ5XNM2LIFD3VD7SPW4J3Z3CWQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 03:19:52.422777702 (515h50m52.293860737s) [noise=true , address=45.91.116.6:4975 , piece_id=2PPMENMABXW2AUIRG6MVFKRMEPEZ5XNM2LIFD3VD7SPW4J3Z3CWQ, seg_pos={0 0}, upload=452621]: Long tail cancellation due to optimal threshold Dec 27 03:19:52.422785907 (515h50m52.293868942s) [noise=true , address=45.91.116.14:4191 , piece_id=YU65LLS4ZE34XS66EQRFLARBBOZ7ZXO2VRQU7I3I6B47IEYEQTLQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 03:19:52.422792420 (515h50m52.293875455s) [noise=true , address=45.91.116.14:4191 , piece_id=YU65LLS4ZE34XS66EQRFLARBBOZ7ZXO2VRQU7I3I6B47IEYEQTLQ, seg_pos={0 0}, upload=452621]: Long tail cancellation due to optimal threshold Dec 27 03:19:52.422807098 (515h50m52.293890133s) [seg_pos={0 0}, upload=452621]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:19:52.422870217 (515h50m52.293953253s) [upload=452621]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:19:52.422874656 (515h50m52.293957691s) [upload=452621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:19:52.709552278 (515h50m52.580635313s) [upload=452621]: Done uploading segments... Dec 27 03:19:52.709651517 (515h50m52.580734562s) [upload=452621]: Upload finished. err: <nil> Duration: 5.712028549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [X....XX.XXX.XXX.XXXX...X.XXXXX.XXXX...X.XX..XX.XXXX..X.X..XX] duration: 30.002876611s trace: https://jaeger.dev.storj.io/trace/14e8898c1be08c07 FAIL (2025-12-27 03:20:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 26YTSAOTHXRYNX23JSUTV2EAHNAXHGRYJ5CDGZ2IJJVBJE6VJQMA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: FTER3HR67PWQL2ZHJBOV7AKYGKJY5TFRROECUFDUJHWRREEF5ADQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: DOYCEVA3VWTXX3PEEN7RFYKNAQE5XJGXANSX5JNUQ2J34QFUQVMA) context deadline exceeded ======LOGS====== Dec 27 03:19:49.188903006 (515h50m49.059986071s) [upload=452621]: Starting upload Dec 27 03:19:49.188985262 (515h50m49.060068297s) [upload=452621]: Done starting upload Dec 27 03:19:49.189079181 (515h50m49.060162226s) [upload=452621]: Uploading segments... Dec 27 03:19:49.189279383 (515h50m49.060362428s) [upload=452621]: Got next segment. Inline: false Dec 27 03:19:49.189319980 (515h50m49.060403015s) [upload=452621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 03:19:49.189325210 (515h50m49.060408265s) [upload=452621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 03:19:49.434468927 (515h50m49.305551982s) [seg_pos={0 0}, upload=452621]: Begin upload segment... Dec 27 03:19:49.434586961 (515h50m49.305670006s) [seg_pos={0 0}, upload=452621]: Done begin upload segment. Dec 27 03:19:49.434622078 (515h50m49.305705133s) [noise=true , address=145.79.227.46:4602 , piece_id=UJDPKL63BUERHIBR7CXHOJF6X2P45LECNDCTH6JDNH3AEOUC45KA, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.434707291 (515h50m49.305790336s) [noise=true , address=45.91.116.9:4263 , piece_id=JANEC6VLCENGC4OEWNFRC23DQH4TFGS6HV5IYHRYQQA2UJ4YKZLA, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.435184499 (515h50m49.306267534s) [noise=true , address=145.79.227.16:4912 , piece_id=RBSEZV6CRPJTL3II5XC2T7RKOFXKF4ZLWY6S4YGFMHG7EORKJT3A, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.435598878 (515h50m49.306681913s) [noise=true , address=145.79.227.17:4970 , piece_id=TAA3JMA2NVNFL7BLDGW5LVFJROJX5GYEJOIMVQFGK3RRLBZZHAMQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.436016293 (515h50m49.307099328s) [noise=true , address=77.90.31.17:4770 , piece_id=INMMZWAQEFSWXNRNLPOMUSHVW4IXAEAM5JDV4PLLRJI6G4QALCQQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.436449328 (515h50m49.307532373s) [noise=true , address=45.91.116.14:4191 , piece_id=YU65LLS4ZE34XS66EQRFLARBBOZ7ZXO2VRQU7I3I6B47IEYEQTLQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.436867955 (515h50m49.307951s) [noise=true , address=45.91.116.8:4046 , piece_id=FTER3HR67PWQL2ZHJBOV7AKYGKJY5TFRROECUFDUJHWRREEF5ADQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.438103697 (515h50m49.309186732s) [noise=true , address=77.90.31.11:4554 , piece_id=6ZM7YLZIHPZXMK4D2XORBXPXN2WWWUE66PQPPJL3OVTZCIUATMPQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.438400583 (515h50m49.309483618s) [noise=true , address=145.79.227.9:4365 , piece_id=MLF3Q7XUPJP2O2AFOXDVT2GD6WFCDAOICO44LQ7FEF4WQKZOVL2Q, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.439157794 (515h50m49.310240829s) [noise=true , address=77.90.31.10:4518 , piece_id=DOYCEVA3VWTXX3PEEN7RFYKNAQE5XJGXANSX5JNUQ2J34QFUQVMA, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.439954161 (515h50m49.311037206s) [noise=true , address=145.79.227.45:4230 , piece_id=IR5WHYBFIJ3YIDK4FLI3WPUZYWIYI7FXOVVCQ4JETORYDHGEESTQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.440203115 (515h50m49.31128615s) [noise=true , address=45.91.116.6:4975 , piece_id=2PPMENMABXW2AUIRG6MVFKRMEPEZ5XNM2LIFD3VD7SPW4J3Z3CWQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.441209411 (515h50m49.312292446s) [noise=true , address=145.79.227.39:4013 , piece_id=237Q46IGU7OI3RQ7OLAEY5ZNW57UFTYCLUKSKGL3O6J2D4XJ3CVQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.442361094 (515h50m49.313444129s) [noise=true , address=145.79.227.37:4565 , piece_id=CD7HGSFMGDUWUUGRX4WX4ICU5VW7ARASTJERCMXE33YC4ZM7U7LA, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.443609141 (515h50m49.314692176s) [noise=true , address=45.91.116.12:4228 , piece_id=26YTSAOTHXRYNX23JSUTV2EAHNAXHGRYJ5CDGZ2IJJVBJE6VJQMA, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:49.443839039 (515h50m49.314922084s) [noise=true , address=145.79.227.35:4481 , piece_id=DMF5XLRTQHKOOFYXT2ATU4N3HCREMCWFF4ZQTQU4FWANR65Q5PYQ, seg_pos={0 0}, upload=452621]: Uploading piece... Dec 27 03:19:51.162583489 (515h50m51.033666554s) [upload=452621]: Next returned nil segment Dec 27 03:19:51.162601392 (515h50m51.033684428s) [upload=452621]: Waiting for error group managing segments... Dec 27 03:19:51.440864473 (515h50m51.311947508s) [noise=true , address=145.79.227.45:4230 , piece_id=IR5WHYBFIJ3YIDK4FLI3WPUZYWIYI7FXOVVCQ4JETORYDHGEESTQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.446548336 (515h50m51.317631391s) [noise=true , address=145.79.227.37:4565 , piece_id=CD7HGSFMGDUWUUGRX4WX4ICU5VW7ARASTJERCMXE33YC4ZM7U7LA, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.452279288 (515h50m51.323362333s) [noise=true , address=145.79.227.39:4013 , piece_id=237Q46IGU7OI3RQ7OLAEY5ZNW57UFTYCLUKSKGL3O6J2D4XJ3CVQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.455079651 (515h50m51.326162696s) [noise=true , address=145.79.227.46:4602 , piece_id=UJDPKL63BUERHIBR7CXHOJF6X2P45LECNDCTH6JDNH3AEOUC45KA, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.455126359 (515h50m51.326209394s) [noise=true , address=145.79.227.17:4970 , piece_id=TAA3JMA2NVNFL7BLDGW5LVFJROJX5GYEJOIMVQFGK3RRLBZZHAMQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.466422419 (515h50m51.337505454s) [noise=true , address=145.79.227.9:4365 , piece_id=MLF3Q7XUPJP2O2AFOXDVT2GD6WFCDAOICO44LQ7FEF4WQKZOVL2Q, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.468365870 (515h50m51.339448905s) [noise=true , address=145.79.227.35:4481 , piece_id=DMF5XLRTQHKOOFYXT2ATU4N3HCREMCWFF4ZQTQU4FWANR65Q5PYQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.477709802 (515h50m51.348792847s) [noise=true , address=145.79.227.16:4912 , piece_id=RBSEZV6CRPJTL3II5XC2T7RKOFXKF4ZLWY6S4YGFMHG7EORKJT3A, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.623148793 (515h50m51.494231838s) [noise=true , address=77.90.31.11:4554 , piece_id=6ZM7YLZIHPZXMK4D2XORBXPXN2WWWUE66PQPPJL3OVTZCIUATMPQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.717006273 (515h50m51.588089318s) [noise=true , address=45.91.116.12:4228 , piece_id=26YTSAOTHXRYNX23JSUTV2EAHNAXHGRYJ5CDGZ2IJJVBJE6VJQMA, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.729552654 (515h50m51.600635689s) [noise=true , address=77.90.31.10:4518 , piece_id=DOYCEVA3VWTXX3PEEN7RFYKNAQE5XJGXANSX5JNUQ2J34QFUQVMA, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:51.814652328 (515h50m51.685735373s) [noise=true , address=45.91.116.8:4046 , piece_id=FTER3HR67PWQL2ZHJBOV7AKYGKJY5TFRROECUFDUJHWRREEF5ADQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:52.422414991 (515h50m52.293498026s) [noise=true , address=45.91.116.9:4263 , piece_id=JANEC6VLCENGC4OEWNFRC23DQH4TFGS6HV5IYHRYQQA2UJ4YKZLA, seg_pos={0 0}, upload=452621]: Done uploading piece. err: <nil> Dec 27 03:19:52.422433807 (515h50m52.293516842s) [seg_pos={0 0}, upload=452621]: Segment reached optimal threshold of 13 pieces. Dec 27 03:19:52.422647394 (515h50m52.293730429s) [noise=true , address=77.90.31.17:4770 , piece_id=INMMZWAQEFSWXNRNLPOMUSHVW4IXAEAM5JDV4PLLRJI6G4QALCQQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 03:19:52.422654206 (515h50m52.293737241s) [noise=true , address=77.90.31.17:4770 , piece_id=INMMZWAQEFSWXNRNLPOMUSHVW4IXAEAM5JDV4PLLRJI6G4QALCQQ, seg_pos={0 0}, upload=452621]: Long tail cancellation due to optimal threshold Dec 27 03:19:52.422756161 (515h50m52.293839196s) [noise=true , address=45.91.116.6:4975 , piece_id=2PPMENMABXW2AUIRG6MVFKRMEPEZ5XNM2LIFD3VD7SPW4J3Z3CWQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 03:19:52.422777702 (515h50m52.293860737s) [noise=true , address=45.91.116.6:4975 , piece_id=2PPMENMABXW2AUIRG6MVFKRMEPEZ5XNM2LIFD3VD7SPW4J3Z3CWQ, seg_pos={0 0}, upload=452621]: Long tail cancellation due to optimal threshold Dec 27 03:19:52.422785907 (515h50m52.293868942s) [noise=true , address=45.91.116.14:4191 , piece_id=YU65LLS4ZE34XS66EQRFLARBBOZ7ZXO2VRQU7I3I6B47IEYEQTLQ, seg_pos={0 0}, upload=452621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 03:19:52.422792420 (515h50m52.293875455s) [noise=true , address=45.91.116.14:4191 , piece_id=YU65LLS4ZE34XS66EQRFLARBBOZ7ZXO2VRQU7I3I6B47IEYEQTLQ, seg_pos={0 0}, upload=452621]: Long tail cancellation due to optimal threshold Dec 27 03:19:52.422807098 (515h50m52.293890133s) [seg_pos={0 0}, upload=452621]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 03:19:52.422870217 (515h50m52.293953253s) [upload=452621]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 03:19:52.422874656 (515h50m52.293957691s) [upload=452621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 03:19:52.709552278 (515h50m52.580635313s) [upload=452621]: Done uploading segments... Dec 27 03:19:52.709651517 (515h50m52.580734562s) [upload=452621]: Upload finished. err: <nil> Duration: 30.002900075s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)